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-^"^In  the  experimental  work,  an  existing  programming  language  tutor  (BIP) 
was  modified  to  allow  students  to  request  and  receive  hints  from  a  human 
tutor.  \The  protocols  generated  during  the  experimental  sessions  were 
analyzed\by  a  taxonomic  system  that  described  the  events  that  took  place 
at  a  sufficiently  general  level  to  allow  comparison  of  sessions  across 
different  subjects  or  different  programming  tasks.  The  actions  of  tho 
student^  the  tutor,  and  the  tutorial  system  itself  were  classified. 


TIhese  experiments  suggested  that:)  ^ 

•  ^Proqrams  are  typically  written  in  two  stages:  layout  of  the 

basic  design  of  the  program  and  (2)  correction  of  the  statements  that 
instantiate  this  design.  c - 

»  Systems  that  rely  on  "canned"  hints  do  not  provide  adequate 
assistance  for  the  complex  problems  faced  by  students*  Although  BIP  was 
able  to  give  reasonable  help  on  the  syntactic  form  of  language  statements, 
it  has  no  facilities  for  Identifying  and  offering  advice  on  problems  with 
a  program's  design;  in  these  experiments,  students  recognized  this  short¬ 
coming  and  relied  upon  assistance  from  the  human  tutors.v«rr 

\ 

The  simple  syntactic  help  offered  by  BIP  and  similar  systems  needs 
to  be  augmented  by  more  powerful  techniques  that  can  understand  the  plan 
underlying  a  student's  program  and  offer  advice  that  corresponds  to  errors 
in  this  plan.  Our  development  of  TURTLE,  a  tutor  for  the  "turtle  graphics" 
component  of  the  LOGO  programming  language,  is  one  step  toward  this  goal. 
TURTLE  uses  artificial  intelligence  techniques  to  identify  and  offer  help  on 
student  plans,  thereby  gaining  the  ability  to  diagnose  the  design  of  a 
student's  program  as  well  as  its  code. 

TURTLE  identifies  student  plans  by  working  within  the  limited  domain 
of  turtle  graphics  and  by  having  access  to  very  detailed  representations 
of  the  problems  posed  to  students.  As  a  result,  it  can  generate  the  entire 
set  of  plausible  solutions  for  a  problem.  Identifying  a  student's  plan  is 
then  reduced  to  the  much  simpler  matter  of  matching  the  student's 
program  to  a  relatively  small  set  of  possible  solutions  and  adopting  the 
the  plan  that  corresponds  to  the  solution  offering  the  best  match. 

Students  are  encouraged  to  decompose  a  more  complex  problem  into  a 
set  of  smaller  problems,  each  of  which  is  solved  by  a  separate  function. 

By  limiting  the  size  of  these  functions,  the  combinatorics  of  solution 
generation  are  kept  under  control,  although  at  the  cost  of  requiring 
TURTLE  to  determine  how  a  student  has  decomposed  a  problem.  This 
determination  is  achieved  in  much  the  same  way  as  the  identification 
of  a  simple  program's  plan:  by  comparing  the  student’s  decomposition  to 
a  known  set  of  possible  and  plausible  decompositions. 

Differences  between  the  student's  proposed  solution  and  the  correct 
solution  --  missing  or  unnecessary  statements,  or  necessary  statements 
with  incorrect  arguments  --  guide  TURTLE'S  hint-generation  component. 

Since  the  student's  plan  is  known,  the  functions  of  the  incorrect  or 
missing  statements  are  also  known,  and  hints  can  be  given  that  reflect 
errors  in  the  design  and  the  code  of  the  student's  program.  The  individu¬ 
alized  and  specific  attention  required  and  requested  by  students  is 
thereby  made  possible. 
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Foreword 

This  is  the  final  report  for  ONR  Contract  No. 
N00014-80-C-0818 ,  "Intelligent  Automated  Tutors  for  Instruction 
in  Planninq  and  Computer  Programming".  The  contract  monitor  for 
this  research  was  Marshall  J.  Farr.  Mark  L.  Miller  was  principal 
investigator  from  October  1980  to  September  1981,  when  he  left 
Texas  Instruments.  Direction  for  the  research  was  then 
transferred  to  Thomas  P.  Kehler  and  James  R.  Miller. 

M.  Miller  was  responsible  for  the  contract  proposal  and  the 
initial  orientation  of  the  research.  OGOL  (Section  2)  was 
designed  by  M.  Miller  and  William  Murray,  and  implemented  by 
Murray.  The  BIP/HINT  system  (Section  3)  was  implemented  (via 
modifications  and  extensions  to  the  original  BIP  code)  by  Patrick 
Duff,  Murray,  and  John  I>.  Shelton.  The  BIP  experiments  were  run 
by  Paul  R.  Michaelis,  Murray,  Hendler,  and  Duff.  The  taxonomic 
analysis  of  the  experiments  (Section  3)  was  done  by  J.  Miller, 
Michaelis,  James  Hendler,  and  Duff.  Murray  designed  and 
implemented  TURTLE  (Section  4) .  The  final  report  was  coordinated 
by  J.  Miller,  with  contributions  from  Kehler,  Michealis,  and 
Murray  (Section  4). 
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Section  1 
Introduction 

Recent  advances  in  computer  technoloqy  have  qreatly  expanded 
the  cower  and  flexibility  of  computer-aided  instruction  (CAI) 
systems.  While  such  systems  have  existed  for  many  years,  the 
development  of  artificial  intelligence  technology  has  allowed 
these  systems  to  become  increasingly  specialized  in  their  control 
of  the  educational  process  and  their  interaction  with  students. 

As  a  result,  these  tutors  offer  a  far  wider  range  of  diagnostic 
and  tutorial  powers  than  is  available  in  tutoring  systems  built 
with  traditional  computer  technology.  A  general  discussion  of 
intelligent  computer-aided  instruction  systems  (ICAI)  can  he 
found  in  Clancey,  Bennett,  and  Cohen  (1981)  and  Sleeman  and  Brown 
(1981) ;  the  current  discussion  will  be  limited  to  CAI  systems 
that  tutor  students  learning  computer  programming  languages. 

One  of  the  first  programming  tutors  was  BIP  (Basic 
Instructional  Program:  Barr,  Beard,  &  Atkinson,  1976).  BIP  offers 
an  environment  in  which  a  student  can  create  and  debug  programs 
written  in  the  BASIC  programming  language.  A  student  working 
with  this  system  is  given  a  series  of  problems  for  which  he  must 
write  oroqrams.  He  enters  the  statements  of  these  programs 
directlv  into  BIP,  which  checks  the  syntax  of  these  statements 
and  advises  the  student  of  any  errors.  BIP  also  provides  a  set 
of  facilities  that  allow  the  student  to  list,  run,  erase,  and 
trace  the  execution  of  the  programs. 

Por  each  of  the  problems  given  to  a  student,  BIP  has  a  set 
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of  input-output  pairs  that  describe  certain  pieces  of  data  and 
the  output  that  should  be  produced  for  these  data  by  the 
student's  program.  It  also  has  a  BASIC  proqram  that  can  generate 
this  output,  qiven  the  data.  The  student  can  run  this  reference 
program,  enter  his  own  data,  and  observe  the  reference  program's 
treatment  of  these  data.  When  the  student  has  refined  his  own 
program  to  the  point  where  he  believes  it  is  correct,  he  can 
submit  it  to  BIP  for  evaluation:  BIP  runs  the  student's  program 
with  the  data  in  its  inout-output  pairs  and  compares  the  output 
of  the  student's  program  to  the  output  that  should  be  produced  by 
these  data.  The  student  is  informed  of  anv  errors  in  this  output 
and  can  use  the  facilities  of  BIP  to  debug  his  program  until  .its 
output  is  correct. 

BIP  has  several  facilities  for  offering  help  to  a  student 
durinq  the  tutoring  session.  The  student  can  request  a  qeneral 
description  of  the  current  problem,  which  summarizes  the  general 
design  of  the  program  and  the  kinds  of  statements  that  are 
necessary  to  solve  the  problem.  The  student  can  also  ask  for  a 
"hint,"  which  is  usually  a  more  detailed  description  of  a 
particularly  difficult  part  of  this  problem.  All  these 
descriptions  are  determined  by  the  instructor  who  defines  the  BIP 
curriculum;  BIP  merely  presents  these  to  the  student  upon 
request.  Finally,  whenever  a  student  enters  a  statement  or  a 
command  that  is  diagnosed  by  BIP  as  beinq  incorrect,  he  is 
offered  help  on  that  statement  or  command.  For  example,  if  a 
student  entered  an  illegal  IF  statement  as  part  of  his  program, 
BIP  would  report  that  the  syntax  of  this  statement  was  incorrect 
and  would  offer  further  help.  If  the  student  requested  this 
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advice,  he  would  receive  a  general  description  of  the  form  of  an 
IP  statement;  further  requests  for  help  would  obtain  examples  of 
other  incorrect  IP  statements  and,  finally,  examples  of  correct 
IP  statements. 

The  level  of  assistance  BIP  can  provide  to  a  student  is 
limited.  While  BIP  can  offer  help  on  the  general  form  of  a 
particular  statement  after  an  incorrect  instance  of  this 
statement  is  entered,  it  can  offer  only  indirect  advice,  via 
references  to  correct  and  flawed  statements  of  the  same  type,  on 
why  that  statement  is  incorrect  and  what  must  be  done  to  correct 
it.  Similarly,  BIP  can  offer  little  help  with  statements  that 
are  syntactically  correct  but  logically  flawed.  Although  BIP  is 
more  than  a  frame-based  tutor,  its  assistance  must  be  predefined 
by  the  coursers  authors  and  written  in  a  general  form  to  address 
general  problems  that  will  presumably  be  of  use  to  as  many 
students  as  possible.  To  go  beyond  these  limitations,  tutorial 
systems  must  incorporate  detailed  knowledge  about  the  actions  the 
program  is  intended  to  carry  out,  the  nature  of  the  programming 
statements  used  to  build  these  programs,  and  high-level  knowledge 
structures  that  describe  the  design  of  a  complex  program;  i.e., 
the  decomposition  of  a  large  problem  into  an  organized  collection 
of  smaller,  manageable  problems.  As  will  be  discussed  later, 
artificial  intelligence  techniques  offer  an  appropriate  set  of 
tools  for  this  task. 

Another  feature  of  BIP  that  might  be  improved  by  artificial 
intelligent  techniques  is  the  selection  of  the  problems  that  are 
presented  to  students.  BIP  teaches  BASIC  by  requiring  the 
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student  to  write  programs  that  accomplish  increasingly  difficult 
tasks.  The  complete  system  contains  91  tasks,  although  only 
about  20  of  these  are  ever  presented  to  a  single  student.  The 
selection  of  these  tasks  deoends  upon  BIP"s  curr iculuni 
information  network :  BIP  contains,  for  each  of  the  problems  in 
the  network,  a  description  of  (a)  the  skills  (statement  types, 
such  as  variable  assignment,  conditional,  input/output,  etc.)  a 
student  must  have  to  solve  this  problem,  (b)  the  skills  the 
problem  is  intended  to  teach,  and  (c)  the  skills  that  are 
considered  irrelevant  to  the  problem.  BIP  also  builds  and 
maintains  a  model  of  the  student  by  noting  the  skills  the  student 
has  mastered  as  a  result  of  completing  the  programming  tasks 
presented  to  him  by  BIP.  When  a  student  has  successfully 
completed  a  task  (and  thus  presumably  mastered  the  newly 
introduced  skills  in  that  task) ,  BIP  adds  the  skills  that  were 
taught  by  this  task  to  its  model  of  the  student,  compares  this 
model  to  its  curriculum  network,  and  presents  the  student  with  a 
task  that  presupposes  a  set  of  skills  that  is  already  possessed 
bv  the  student  and  that  introduces  or  develops  a  skill  that  has 
not  been  mastered  by  the  student. 

In  this  way,  BIP  can  tailor  its  instruction  to  the  progress 
of  individual  students,  presenting  tasks  that  gradually  advance 
the  student  toward  a  complete  understanding  of  BASIC.  Note, 
however,  that  BIP's  lack  of  knowledge  structures  that  describe 
the  design-level  components  of  a  program  (which  prevents  BIP  from 
offering  students  assistance  at  this  level)  also  limits  its 
ability  to  build  a  detailed  and  accurate  student  models  students 
learning  to  program  are  learning  software  design  as  well  as  a 


particular  programming  language,  and  a  sound  tutorial  system 
should  represent  both  of  these  aspects  of  the  students" 
experience. 

These  issues  have  been  considered  in  more  recent  tutorial 
systems  that  have  taken  advantage  of  insights  from  artificial 
intelligence  research.  To  provide  personalized  advice  about  the 
desiqn  of  a  student's  program  and  about  the  actions  carried  out 
by  individual  statements,  it  is  necessary  to  examine  the  content 
of  the  student's  partially  correct  program,  to  infer  the  plan 
underlying  this  program,  and  to  offer  assistance  to  the  student 
that  addresses  the  differences  between  this  inferred  plan  and  a 
plan  that  describes  a  correct  solution  to  the  problem. 

One  system  concerned  with  inferring  student  plans  to  this 
end  was  Goldstein's  MYCROFT  (1975) ,  a  system  designed  to  diagnose 
figures  drawn  by  "turtle  graphics"  programs  written  in  Logo.  A 
student  working  with  MYCROFT  describes  the  figure  he  was  trying 
to  draw  in  a  "model  lanquaqe";  a  stick  figure  of  a  man  might  be 
described  as: 

MODEL  MAN 

Ml  PARTS  HEAD  BODY  ARMS  LEGS 
M2  EQUTRI  HEAD 
M3  LINE  BODY 
M4  V  ARMS,  V  LEGS 

M5  CONNECTED  HEAD  BODY,  CONNECTED  BODY  ARMS, 

CONNECTED  BODY  LEGS 
M6  BELOW  LEGS  ARMS,  BELOW  ARMS  HEAD 
END 

This  description  states  that  a  MAN  has  a  HEAD,  BODY,  ARMS, 
and  LEGS  as  its  PARTS  (Ml) .  The  tvpes  of  these  figures  are  then 
identified  (e.g,  the  HEAD  is  an  EQUTRI  —  an  equilateral  triangle 
[M2]),  as  are  the  connections  between  these  parts  (M5)  and  the 


relative  positions  of  the  parts  (M6) .  Some  of  these  property 
types  were  defined  as  system  primitives  (e.g.,  PARTS,  LINE, 
CONNECTED,  BELOW)  while  other  properties  could  be  defined  by  the 
student:  here,  the  student  must  define  EQUTRI  and  V,  in  ways 
similar  to  the  definition  of  MAN. 

MYCROFT  could  use  this  figure  description  to  diagnose  errors 
in  the  student's  program  in  terms  of  high-level  principles 
underlying  the  design  of  the  program,  such  as  the  failure  to 
reorient  the  "turtle"  drawinq  device  after  drawing  a  portion  of 
the  figure,  or  that  an  error  existed  in  the  statements  that  drew 
the  LEGS  of  the  MAN.  The  rules  that  MYCROFT  used  to  detect 
programming  errors  were  ad  hoc,  but  captured  many  of  the  common 
flaws  in  program  construction.  In  addition,  MYCROFT  was  not  fully 
implemented . 

Similar  concerns  for  assisting  students  with  the  design  of 

their  proqrams  lay  behind  the  development  of  SPADE  (Miller,  1979; 

Miller  &  Goldstein,  1977) .  SPADE  guided  a  student  through  the 

construction  of  a  Logo  oroqram  by  providing  a  high-level 

description  of  the  design  of  the  program  that  was  progressively 

refined  by  the  student.  SPADE  used  grammar-like  structures  to 

represent  possible  design  choices: 

PLAN  ->  IDENTIFY  |  DECOMPOSE  I  REFORMULATE 

IDENTIFY  ->  PRIMITIVE  |  DEFINED 
DECOMPOSE  ->  CONJUNCTION  |  REPETITION 

•  •  • 

PRIMITIVE  ->  VECTOR  |  ROTATION  |  PENSTATE 
VECTOR  ->  {FORWARD  |  BACK}  +  <number > 

A  student  using  SPADE  (and  the  design  grammar  shown  here)  is 
presented  with  the  decomposition  of  PLAN,  indicating  that  a 


program  can  be  written  by  identifying  primitive  or  previously 
defined  procedures  that  would  accomplish  the  task,  by  decomposing 
the  problem  into  a  set  of  simpler  tasks,  or  by  reformulating  a 
problem  into  a  different  problem  for  which  a  known  plan  exists. 
After  specifying  the  desired  decomposition,  The  student  continues 
by  selecting  one  of  these  decompositions  for  further  work, 
perhaps  DECOMPOSE:  by  this,  the  student  indicates  that  the 
program  required  a  decomposition  (although  he  can  later  return  to 
the  PLAN  level  and  work  on  the  IDENTIFY  or  REFORMULATE  steps, 
should  these  be  relevant) .  The  student  is  then  shown  the 
decomposition  of  the  DECOMPOSE  step  and  further  decides  whether 
the  required  decomposition  is  one  of  CONJUNCTION  or  REPETITION. 

As  this  process  continues,  the  student  ultimately  reaches  the 
terminal  points  of  the  grammar,  which  specified  explicit 
programming  language  statements.  The  result  of  this  attention  to 
the  design  component  of  programming  is  meant  to  convey  basic 
concepts  of  software  design  rather  than  the  syntax  of  the 
programming  language. 

A  somewhat  different  approach  to  these  issues  was  taken  by 
Centner's  FLOW  tutor  (Gentner,  1977;  Gentner  &  Norman,  1977). 

This  system  contained  schematic  structures  that  described  a  wide 
variety  of  states  that  might  be  present  during  the  construction 
of  a  program,  from  individual  keystrokes  to  complex  design-level 
structures.  The  system  began  its  control  of  a  tutoring  session  by 
locating  the  schema  that  matched  the  problem  posed  to  a  student 
and  expanding  this  schema  into  instances  of  lower-level  schemata, 
predicting  individual  statements  and  even  keystrokes.  The 
correctness  of  a  student's  progress  toward  a  solution  of  the 


problem  could  then  be  tracked  by  noting  which  schemata  were 
activated  by  the  student's  program.  If  the  student  proceeded 
toward  a  correct  solution  of  the  problem,  the  schemata  activated 
by  the  student's  oroqram  would  match  those  activated  by  the 
tutor's  decomposition  of  the  oroblem  description.  Alternative 
correct  solutions  to  a  problem  would  be  correctly  interpreted  by 
the  system,  since  a  correct  solution  would  satisfy  the 
constraints  of  high  level  schemata,  even  though  the  inC’"vidual 
proqram  statements  would  have  activated  an  unexpected  set  of 
oroqram  statement  schemata.  If  the  student's  program  departed 
significantly  from  a  correct  solution,  the  resulting  errors  would 
activate  other  schemata  that  were  designed  to  recognize  these 
errors,  which  would  then  trigger  appropriate  corrective  actions. 
It  is  important  to  note  that  these  error-detecting  schemata  could 
be  constructed  so  as  to  describe  and  detect  errors  at  any  level 
of  complexitv,  from  an  assignment  statement  with  unbalanced 
parentheses  to  the  incorrect  initialization  of  a  counter  variable 
in  a  complex  iterative  loot). 

The  present  research  has  focused  on  one  of  these  problems  — 
how  to  give  students  specific,  individualized  assistance  durinq 
the  construction  of  computer  programs.  We  proposed  to  construct 
a  tutor  that  would  teach  students  how  to  write  proqrams  that 
oroduc?  "turtle  qraphics"  line  drawings,  and  that  would  offer  the 
students  hints  about  errors  in  their  program.  The  proposed  tutor 
was  intended  to  address  not  only  syntactic  errors  in  program 
statements,  but  also  errors  in  the  conceptual  design  of  the 
oroqram.  This  research  was  intended  to  focus  on  one  aspect  of 
the  hint-qivinq  problem:  the  construction  and  presentation  of 
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relevant  hints.  We  have  postponed  until  a  later  time  the  second 
aspect  of  this  problem  —  determining  when  the  tutor  should 
interrupt  the  student "“s  work  to  offer  a  hint  —  by  requiring  the 
student  to  explicitly  request  hints  from  the  tutorial  system. 

This  research  is  presented  in  four  sections.  First,  a 
programming  language  for  studying  tutorial  interactions  and  a 
simple  tutor  for  this  language  are  described.  Second,  a  series 
of  experiments  on  a  computer-based  tutor  with  human-qenerated 
hints  is  presented.  Third,  the  tutoring  system  that  was 
implemented  is  described  and  demonstrated.  Fourth,  the  research 
is  summarized  and  evaluated,  and  directions  for  future  research 


are  considered 


Section  2 


OGOL:  A  Graphics-Oriented  Language  for  Tutorial  Systems 

2.1  Task  Definition  and  Language  Construction 

The  first  step  in  this  project  was  to  identify  a  programming 
lanquage  that  would  be  particularly  appropriate  for  the 
development  of  a  tutorial  system.  The  primary  candidates  for 
this  language  were  Loqo  and  Lisp.  Loqo-based  "turtle  graphics" 
programs  are  easy  to  explain  to  students,  can  be  made  arbitrarily 
complex,  and  result  in  a  drawing  in  which  the  correct  and 
incorrect  oarts  of  a  proqram  are  quite  apparent.  In  addition, 
these  drawinqs  can  be  represented  by  networks  of  points  and 
vectors,  a  convenient  data  format  for  those  oarts  of  the  tutorial 
system  that  must  analyze  and  evaluate  student  programs.  However, 
the  limited  power  and  generality  of  Logo's  iterative  and 
conditional  branchinq  statements  would  make  the  construction  of 
proqrams  that  require  these  statements  inordinately  difficult  and 
would  limit  the  extension  of  a  Loqo-based  tutorial  system  to 
other  lanquages  in  which  more  powerful  control  statements  are 
available.  In  contrast,  Lisp  offers  a  straightforward  solution 
to  the  extensibility  problem;  however,  it  lacks  the  qraphics 
statements  that  made  Loqo  so  appeal inq  and  would  impose  upon  the 
student  a  set  of  function  names  and  programming  structures  that 
are  often  difficult  and  confusing  for  those  who  are  new  to 
programming  or  who  are  experienced  with  Fortran-  or  Pascal-like 
lanquaqes. 

This  dilemma  was  resolved  by  developing  a  new  language  — 
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OGOL  (ONR  Graphics  Oriented  Language)  —  that  merges  the 
desirable  features  of  Logo  and  Lisp.  In  particular,  OGOL  allows 
the  construction  of  turtle  graphics  programs  in  an  environment 
with  all  the  extensibility  and  recursive  power  of  Lisp  and  with 
several  features  that  correct  some  of  the  more  complex  and 
difficult  to  learn  aspects  of  Lisp:  OGOL  offers  flexible  list 
construction  facilities,  pattern  directed  evaluation,  and  both 
call-by-name  and  call-by-value  evaluation. 

2.2  An  OGOL  Tutorial  System 

To  evaluate  the  usefulness  of  this  language,  a  simple 
"turtle  graphics"  tutor  was  built  around  OGOL.  The  Lisp-like 
features  of  OGOL  were  not  studied  in  this  system.  Rather,  the 
OGOL  tutor  trains  the  student  in  the  use  of  the  basic  turtle 
graphics  commands.  This  tutor  uses  two  terminals:  a  standard 
terminal  with  which  the  student  carries  on  the  tutorial 
interaction  with  the  system,  and  a  color  graphics  terminal  that 
displays  the  graphics  commands  and  programs  run  by  the  student  or 
the  tutorial  system.  The  graphics  commands  available  in  OGOL 
are : 

-ERASE:  erase  the  screen  and  place  the  turtle  in  the 
center  of  the  screen. 

-DRAW  X:  draw  a  line  X  steps  long. 

-MOVE  X:  move  the  turtle  forward  X  steps  (without 
drawing  a  line) . 

-TURN  X:  turn  the  turtle  X  degrees  clockwise. 

This  tutor  is  in  many  ways  similar  to  BIP.  Like  BIP,  the 
OGOL  tutor  uses  a  "curriculum  information  network"  to  expose  a 
student  to  the  course  material  in  an  ordered  way.  The  OGOL  tutor 
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also  has  a  facility  for  offering  predetermined  hints  about  the 
tasks  being  presented  to  the  student.  The  instructor  who 
assembles  a  particular  version  of  the  OGOL  tutor  can  specify  a 
set  of  hints  for  each  proqram  in  the  curriculum.  These  hints  can 
refer  to  either  the  general  design  of  the  program  being  written 
or  to  the  code  used  in  specific  statements  within  the  program. 
Upon  request,  these  hints  will  be  shown  one  after  another  in  the 
order  specified.  The  presentation  of  these  hints  is  partially 
affected  by  the  student's  progress  in  the  task:  if  the  student 
asks  for  hints  before  he  has  tried  to  enter  a  program  that  would 
solve  the  current  problem,  he  will  receive  the  design-level  hints 
provided  by  the  instructor.  However,  he  cannot  receive  code 
hints  until  he  has  made  at  least  one  attempt  to  define  the 
program. 

A  partial  transcript  of  a  student's  interaction  with  the 
OGOL  tutor  is  shown  in  Table  1.  The  session  begins  by  presenting 
some  introductory  information  about  computer  programming  in 
qeneral  and  turtle  graphics  programming  in  particular,  and  by 
allowing  the  student  to  experiment  with  the  basic  OGOL  graphics 
commands  —  the  student  can  issue  graphics  statements  and  observe 
their  result  on  the  graphics  terminal.  In  the  transcript  in  Table 
1,  the  student  has  completed  this  phase  of  the  session  and  is 
receiving  instruction  on  how  to  generate  a  sequence  of  OGOL 
statements  that  will  draw  a  square. 

The  student  is  first  told  that  he  can  draw  a  square  by 
putting  the  individual  commands  he  previously  used  to  draw  a 
square  into  a  proqram  definition  statement  (line  1).  The  tutor 
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Table  1 


A  sample  tutorial  session  with  the  OGOL  tutor. 
Those  lines  entered  by  the  student  (such  as  line  15) 

are  headed  by 
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Let's  write  a  program  to  draw  a  square.  You 
already  know  how  to  do  this  by  tvpinq  in  a  sequence  of 
commands.  Write  a  program  named  SQUARE  that  will  perform 
that  sequence  of  commands  when  it  is  called.  Make  sure 
that  the  heading  of  the  turtle  before  and  after  calling 
the  SQUARE  program  is  the  same.  This  may  require 
another  TURN  command  at  the  end  of  the  sequence  of 
commands  to  draw  the  square.  This  final  turn  command 
will  make  sure  that  the  sum  of  all  the  rotations  in  the 
SQUARE  program  is  360. 

Welcome  to  OGOL1 
Type  (HELP)  for  help. 

**  (hint) 

If  I  wanted  to  define  a  program  called  FOO  to  have  my 
turtle  draw  forward  100  units,  turn  135  degrees,  then 
draw  50  units,  I  would  type  the  followings 

(define  [foo] 

(draw  100) 

(turn  135) 

(draw  50)) 

Note  that  I  can  place  the  various  parts  of  the  program  on 
different  lines.  When  all  the  parentheses  balance  [that 
is  when  the  number  of  left  parentheses:  (  matches  the 
number  of  right  parentheses:  )  ]  and  I  have  entered  a 
return  after  the  last  line,  then  the  proqram  is  defined. 

To  call  FOO,  I  need  only  type: 

(FOO) . 

** (hint) 

Sorrv,  no  more  hints  are  available  for  designing  a 
solution  to  this  task.  However,  I  can  give  you  more 
hints  once  you  DEFINE  a  proqram. 

**  (define  [square] 

(draw  100)  (draw  100)  (draw  100)  (draw  100)) 

SQUARE 
**  (hint) 


Your  program  should  call  the  same  sequence  of  commands 
you  called  before  when  you  drew  4  line  segments  at  right 
angles  to  each  other.  If  you  make  an  error,  you  can 
always  redefine  your  program  and  the  new  definition  will 
override  the  old  definition.  If  your  sequence  of 
commands  turns  the  turtle  270  degrees,  then  adding  (TURN 
90)  at  the  end  of  that  sequence  will  restore  the 
original  heading  of  the  turtle. 

** (define  [square]  (draw  100)  (turn  90) 

(draw  100)  (turn  90) 

(draw  100)  (turn  90) 

(draw  100)  (turn  90)) 

SQUARE 


** (square) 

[[  OGOL  draws  a  square  on  the  graphics  terminal  ]] 

** (quit) 

Here  is  one  way  to  solve  the  task  you  have  just 
completed : 

(define  [square] 

(draw  100) 

(turn  90) 

(draw  100) 

(turn  90) 

(draw  100) 

(turn  90) 

(draw  100) 

(turn  90)) 

Note  the  last  (Turn  90)  is  used  to  return  the  turtle  to 
its  original  heading. 

See  if  you  can  draw  a  triangle  now,  in  a  similar 
fashion  to  the  way  you  drew  the  square,  using  only  DRAW 
and  TURN  commands.  The  triangle  should  be  equilateral 
and  have  sides  of  length  100.  It's  OK  if  the  triangle 
is  tilted. 


[session  continues] 
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then  places  the  student  in  the  OGOL  interpreter  (line  12) ,  where 
he  is  to  define  the  SQUARE  function. 

The  student  begins  this  phase  of  the  session  by  requesting  a 
hint  from  the  tutor  (line  15) ,  which  provides  a  hint  about  the 
design  of  the  DEFINE  statement:  he  is  shown  how  to  define  a 
function  that  draws  an  arbitrary  line  segment  (line  21).  The 
student  requests  another  hint  (line  35) ,  but  since  the  student 
has  not  vet  tried  to  define  the  SQUARE  function,  and  since  only 
one  design  hint  was  included  in  the  system  by  the  instructor,  he 
is  told  that  no  more  hints  will  be  available  until  he  has  tried 
to  define  the  function  (line  37). 

At  this  point,  the  student  (incorrectly)  defines  SQUARE 
(line  41).  His  subsequent  request  for  a  hint  is  answered  (line 
47)  with  a  reference  to  the  series  of  commands  he  had  entered 
earlier.  He  then  (correctly)  redefines  SQUARE,  (line  56)  and  runs 
the  function  (line  62) ,  which  draws  an  acceDtable  square  on  the 
qraohics  terminal.  Having  completed  the  function,  he  leaves  the 
OGOL  tutor  (line  66)  and  returns  to  the  tutor,  which  shows  him  a 
valid  function  for  SQUARE  (line  71) .  The  session  then  continues, 
with  the  student  now  requested  to  define  a  function  to  draw  a 
trianqle  (line  94) . 

2.3  Evaluation 

As  stated  earlier,  the  OGOL  tutor  is  quite  primitive.  All 
hints  are  prespecified  by  a  human  instructor,  and  its  curriculum 
network  was  never  developed  to  the  extent  of  the  corresponding 
network  in  BIP.  Further,  this  system  was  never  extended  toward 
becoming  a  trulv  intelligent  tutorial  system,  nor  even  toward 
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acquiring  the  full  capabilities  of  a  BIP-like  system.  As  a 
result  of  some  preliminary  experiments  with  OGOL,  it  became  clear 
that  the  basic  turtle  graphics  domain  itself  — without  OGOL's 
expanded  facilities  for  subroutine  parameter  passing  and  flow  of 
control  —  was  sufficiently  complex  for  the  purposes  of  the 
contract.  Hence,  the  development  of  the  OGOL  tutor  was  suspended 
in  favor  of  developing  a  turtle  graphics  system  capable  of 
identifying  the  student's  plan  and  generating  intelligent  hints 
about  that  plan. 
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Section  3 

BIP/HINT:  Experiments  with  a  Tutor  with  Human-Generated  Hints 
3.1  The  Tutorial  System 

An  important  part  of  this  study  of  intelligent  tutorial 
systems  has  been  to  study  how  human  tutors  interact  with  their 
students.  Good  tutors  can  build  sound  models  of  their  students' 
knowledge  and  problem  solving  strategies  and  can  interact  with 
the  student  in  ways  that  address  and  correct  the  difficulties  he 
faced.  Identifying  the  strategies  that  good  human  tutors  use  to 
these  ends  is  the  first  step  in  the  construction  of  tutorial 
systems  with  similar  capabilities. 

To  this  end  —  understanding  the  relative  strengths  and 
weaknesses  of  human  and  computer  tutors  —  experiments  were 
carried  out  with  a  modified  version  of  BIP  (Barr,  et  al.,  1975). 
These  modifications  allowed  a  student  in  a  tutoring  session  to 
request  a  hint  from  a  human  tutor.  In  this  modified  version  of 
BIP  (here  called  BIP/HINT)  the  student  and  the  tutor  were  located 
in  separate  rooms,  each  equipped  with  two  terminals  (see  Figure 
1).  The  student  used  one  of  the  terminals  to  interact  with 
BIP/HINT;  both  sides  of  this  interaction  also  appeared  on  one  of 
the  tutor's  terminals.  At  any  time  during  the  session  the 
student  could  request  a  hint  from  the  human  tutor  by  pressing  the 
touch  panel  mounted  on  his  second  terminal.  The  tutor  entered 
this  hint  on  his  second  terminal,  which  then  appeared  on  the 
student's  second  terminal.  This  procedure  was  intended  to 
simulate  a  tutorial  system  with  a  "hint  button";  in  these 


experiments,  the  student  had  to  request  a  hint  (the  tutor  could 
not  interrupt  the  student's  session  and  offer  a  hint) ,  and  the 
student  could  communicate  with  the  tutor  only  by  pressing  the 
touch  panel  (the  student  could  not  send  a  question  to  the  tutor) . 

Some  other  modifications  were  made  to  make  BIP  more  suitable 
for  the  current  purposes: 


*  Session  recordings.  A  student's  interactions  with  BIP/HINT 
and  with  his  tutor  were  saved  in  disk  files  for  later  analysis. 


*  Restricted  set  of  tasks.  Only  seven  of  BIP's  original 
tasks  were  used  in  these  experiments: 

-GREENFLAG:  an  introduction  to  the  use  of  BIP  and  the 
format  of  BASIC  statements,  and  the  quided  construction 
and  execution  of  a  BASIC  proqram  that  assiqns  an 
integer  value  to  a  variable  and  then  prints  the  value 
of  that  integer. 

-ARTICHOKE:  assign  the  string  "ARTICHOKE"  to  a  string 
variable,  assign  the  value  of  that  variable  to  a  second 
variable,  and  print  the  second  variable. 

-SINOP:  get  two  strinqs  from  the  student  with  two 
separate  INPUT  statements  and  print  them  on  the  same 
line. 

-NINOP:  qet  two  numbers  from  the  student  and  print  their 
sum,  difference,  product,  and  quotient. 

-TWOS:  use  a  FOR  loop  to  qenerate  and  print  the  even 
numbers  from  2  to  some  number  specified  by  the  student. 

-NGREAT:  qet  two  numbers  from  the  student  and  print  them 
in  the  proper  order  in  the  form,  "<number>  is  greater 
than  <number>". 

-CALCULATOR:  qet  an  arithmetic  operation  from  the 
student  (l«"add,"  2="subtract , "  etc.)  and  apply  this 
operation  to  two  numbers  that  are  then  entered  by  the 
student. 


Some  minor  changes  were  made  in  these  tasks  to  simplify  the 
wordings  of  the  problems  and  to  remove  a  few  ambiguities. 


STUDENT 


TERMINAL  1 
/N 


noraal 
BIP 
sess i on 


TERMINAL  ~ 


B  I  P  /  H  I  N  T 


copy  of 
student's 
BIP 
sess i on 


TERMINRL  1 


TUTOR 

Figure  1  A  Schematic  Description  of  the  Experimental  islIViiiN 
System 
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*  Restricted  set  of  BIP  commands  and  BASIC  statements. 
Students  workinq  with  BIP  can  he  shown  the  names  of  the  leqal  BIP 
commands  and  BASIC  statements  by  usinq  the  commands  "?BIP"  and 
"7BASIC",  respectively.  Both  commands  produce  larqe  amounts  of 
information,  and  some  students  complained  about  being  unable  to 
find  a  command  or  statement  in  these  lists.  Because  of  this  and 
the  restricted  set  of  tasks  used  in  BIP/HINT,  some  of  the 
commands  and  statements  were  removed  from  BIP/HINT's  response  to 
the  "?"  queries,  although  the  parts  of  BIP  responsible  for 
interpretinq  the  disabled  commands  and  statements  remained 
intact,  so  that  these  statements  would  be  interpreted  properly  if 
they  were  entered.  The  commands  and  statements  available  in 
BIP/HINT  were  then  as  follows: 

Available  BIP  commands:  CALC,  DEMO,  LIST,  MORE,  RUN,  SCR, 
WHAT,  WHO,  WHY 

Hidden  BIP  commands:  DEMO-TRACE,  FLOW,  HINT,  KILL,  MORE, 

BYE 

Available  BASIC  statements:  DATA,  END,  FOR,  GOTO,  IF, 

INPUT,  LET,  NEXT,  PRINT,  READ,  REM,  STOP. 

Hidden  BASIC  statements:  BEGINSUB,  DIM,  ENDSUB,  GOSIJB, 
REOPEN,  RETURN 

*  Revised  student  manual .  The  BIP  student  manual  was  revised 
to  make  it  easier  to  locate  desired  parts  of  the  manual  and  to 
remove  those  parts  of  the  manual  that  referred  to  BIP  commands 
that  were  not  available  in  BIP/HINT.  This  revised  manual 
included  selected  sections  of  the  oriqinal  BIP  manual  and  new 
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one-page  summaries  of  the  most  common  BIP  commands  and  BASIC 
statements.  These  summaries  contained  brief  descriptions  of  the 
command  or  statement  and  the  page  number  of  the  manual  where  a 
more  detailed  description  could  be  found.  This  revised  manual 
covered  all  the  legal  BASIC  statements,  but  was  only  37  pages 
long  (versus  the  61  pages  of  the  oriqinal  manual) . 

3.2  Experimentation  and  Protocol  Analysis  Taxonomy 

The  experimental  sessions  were  conducted  to  study  the 
conditions  under  which  students  requested  assistance  from  the 
human  tutor  or  from  the  tutoring  system,  the  nature  of  these 
hints  and  the  tutoring  experience  in  general,  and  the  ability  of 
the  student  to  apply  these  hints  toward  the  construction  of  a 
correct  proqram.  There  are  important  differences  between  BASIC 
and  LOGO,  but  our  analyses  of  these  protocols  are  not  concerned 
with  the  lanquaqe-deoendent  aspects  of  the  tutoring  sessions. 
Rather,  thev  are  focused  on  the  cognitive  aspects  of  tutor inq, 
especially  those  related  to  the  circumstances  surrounding  hint 
requests  and  hint  generation.  It  should  also  be  noted  that  these 
experiments  should  not  be  regarded  as  an  evaluation  of  BIP.  The 
version  of  BIP  used  in  these  experiments  was  severely  limited  in 
comparison  to  the  actual  system  —  in  particular,  BIP"s  own  hint 
facility  was  hidden  from  the  students.  The  intent  was  only  to 
observe  the  nature  of  the  interaction  between  a  student  and  a 
tutor . 


Eiqht  subjects,  none  of  whom  had  any  prior  experience  with 
computer  proqramminq,  took  part  in  a  series  of  sessions  with 
BIP/HINT.  Due  to  problems  with  the  computer  system  (e.g.. 


BIP/HINT  crashes  durinq  a  tutorial  session) ,  we  have  usable 
protocols  from  three  of  these  subjects.  All  subjects 
successfully  completed  the  first  five  tasks:  GREENFLAG, 
ARTICHOKE,  SINOP,  NINOP,  and  TWOS. 

Because  the  protocols  are  very  lonq,  it  was  necessary  to 
condense  them  into  a  form  more  amenable  to  analysis.  A  taxonomy 
was  therefore  devised  to  describe  the  interactions  by  the 
student,  the  tutor,  and  BIP/HINT  in  these  protocols  (Table  2). 

The  taxonomy  contains  three  major  classifications, 
correspond inq  to  whether  the  protocol  segment  beinq  encoded  was 
produced  by  the  student,  by  the  human  tutor ,  or  by  the  tutorial 
system.  A  student  may  do  a  variety  of  thinqs  to  oroqram 
statements:  he  mav  try  to  enter  a  line  in  the  reference  proqram 

for  the  first  time,  reenter  that  statement  until  a  syntactically 
leqal  BASIC  statement  has  been  entered,  change  that  leqal 
statement  further,  or  delete  the  a  statement  altogether.  He  may 
also  enter  various  BIP  commands  or  request  help  from  either  the 
tutor  (by  pressing  the  touch  panel)  or  the  tutorial  svstem  (in 
response  to  an  offer  of  help,  as  described  below) .  The  student 
mav  also  respond  to  one  of  a  number  of  questions  from  BIP. 

1 

Our  tutors"  statements  fell  into  two  categories:  offers  of 
future  help  and  hints  of  various  kinds.  We  identified  four  major 
classes  of  hints: 

-code:  hints  that  included  examples  of  executable  BASIC 

statements . 

_ 

Several  people  served  as  tutors  in  these  experiments. 


Table  2 


Taxonomy  for  the  analysis  of  BIP  protocols 


student 


enters 

command  <name  <args>> 
data 

statement: 

line  <BIP-line#> 

reenters 


chanqes 


deletes 


statement: 

line  <BIP-line#> 

statement: 

line  <BIP-line#> 


statement: 
line  <BIP-Iine#> 
evaluates 

proqram: 

ok:  incorrect] 

wrong:  {incorrect} 
answers  BIP  querv  <with  response> 
asks  for 

system 

{no}  help 

tutor 

hint 


tutor 

qives  hint: 

code : 

{ok  |  error  |  continue} 
line  <BIP-line#> 

design: 

{ok  |  error  |  continue} 
line  <BIP-line#> 
operation: 

{ok  |  error  |  continue} 

general: 

{ok  |  error  I  continue} 


offers 


future  help 
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system 

accepts 

command  <name> 

modification:  line  <BIP-Iine#> 
statement:  line  <BIP-line#> 
asks  for  operation  instructions 
describes  execution: 

fails:  line  <BIP-line>,  error  <BIP-error> 

incut  error 

runs 

ok 

wrong 

qives  hint 

code : 

line  <BIP-line#>, 
continue 
error 
ok 

design : 

line  <BIP-line#>, 
continue 
error 
ok 

operation : 

continue 

error 

ok 

general: 

continue 

error 

ok 

offers 

help: 

line  <BIP-line#> 
operation 

no  help 

problem  description 

reiects 

command:  <name  <arqs>> 

has  a  line  number 
issued  at  wrong  time 
incomplete  command 
other  error 

statement:  line  <BIP-line#>, 
has  no  line  number 
incomplete-statement 
other  error 

syntax  error:  <BIP-error> 


terminates  task 
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-design:  general  hints  about  the  design  of  the  program, 
or  hints  that  suggested  the  use  of  particular  BASIC 
statements,  but  that  did  not  show  an  executable 
statement.  Hence,  the  hint,  "Use  a  PRINT  statement  to 
output  the  value  of  X"  would  be  scored  as  a  design 
hint,  while  "Use  the  statement  "PRINT  X"  to  output  the 
value  of  X"  would  be  scored  as  a  code  hint. 

-operation:  hints  about  the  operation  of  BIP  (e.g.,  "Use 
the  LIST  command  to  look  at  your  program."). 

-general:  hints  that  referred  to  general  aspects  of 
computer  programming  (e.g.,  "The  computer  can  only  do 
exactly  what  vou  tell  it  to  do.") 


In  addition,  the  scoring  of  the  hint  noted  the  line  of  the 
reference  program  with  which  the  hint  was  concerned,  and  whether, 
through  this  hint,  the  tutor  was  telling  the  student  that  an 
error  had  been  made,  that  the  relevant  part  of  the  program  was 
correct  (ok) ,  or  that  the  student  should  simply  continue  working. 

The  oarts  of  the  protocol  generated  by  the  tutorial  system 
were  scored  in  similar  ways.  The  system's  acceptance  or 
rejection  of  the  BASIC  statements  and  BIP  commands  entered  by  a 
student  were  scored  appropriately.  BIP  could  also  offer  help 
after  a  statement  or  command  error,  ask  the  student  for  certain 
operation  instructions,  and  describe  the  execution  of  the 
student's  proqram.  These  diagnostic  messages  and  problem 
descriptions  typically  contained  suggestions  about  how  the 
proqram  should  be  constructed,  and  so  were  scored  the  same  wav  as 
were  tutor  hints. 


This  taxonomic  system  converts  the  near-natural  lanquage  of 
the  tutorial  session  into  a  more  limited  form,  which  offers 
several  advantages  now  and  others  that  might  be  explored  in  the 
future.  The  use  of  this  taxonomy  has  allowed  us  to  automate  much 
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of  the  descriptive  analysis  of  these  protocols,  and  it  serves  as 
a  common  lanquage  through  which  we  can  compare  the  oerformance  of 
all  the  subjects  on  a  common  task,  or  evaluate  the  progress  of 
individual  subjects  on  successive  tasks.  A  sample  protocol  and 
its  taxonomic  analysis  are  shown  in  Table  3.  Statements  from  BIP 
are  in  upper  case,  responses  from  the  student  are  in  lower  case; 
the  taxonomic  analyses  of  the  statements  follow  exclamation 
points,  and  the  tutor's  hints  are  underlined. 

The  session  shown  in  this  protocol  began  with  an  offer  by 

BIP  to  print  a  description  of  the  current  task  (lines  1-2) ;  this 

was  coded  as  "system  offers  problem  description"  (line  3) .  By 

responding  "yes"  (line  5),  the  "student  answers  [a]  BIP  query." 

This  description  of  the  problem  contained  several  hints  of 

different  kinds.  First,  the  student  was  shown  two  explicit 

examples  of  how  variables  can  be  assigned  the  values  of  string 

constants  or  other  string  variables  (lines  9  and  15).  Since 

these  "system  hints"  contained  instances  of  executable  BASIC 

code,  thev  were  further  classified  as  code  hints  about  specific 

2 

lines  of  BIP's  reference  proqram  for  this  problem.  The  student 
also  received  other,  more  abstract  hints  about  the  design  of  this 
particular  program;  these  (lines  19,  23,  and  26)  were  textual 
descriptions  of  what  each  line  of  this  program  should  do,  and 

- 

The  BIP  tasks  we  have  studied  have  been  simple  enough  that  our 
students'  programs  rarely  differed  in  any  substantial  ways  from 
these  reference  programs.  As  a  result,  we  were  able  to  use  BIP's 
reference  program  as  a  standard  against  which  the  students' 
programs  were  scored:  the  program  statements  entered  by  students 
were  encoded  in  terms  of  the  corresponding  statement  in  BIP's 
reference  program.  This  use  of  a  standard  reference  program 
would  probably  not  be  satisfactory  with  more  complex  programs. 
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Table  3: 

A  sample  application  of  the  taxonomy  to  a  RIP  protocol. 
BIP's  exchanges  are  in  upper  case,  the  student's  responses 
are  in  lower  case,  the  tutor's  hints  are  underlined,  and 
the  taxonomic  statements  are  to  the  right  of  an  exclamation 

point. 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 


YOU  ARE  IN  TASK  ARTICHOKE. 

DO  YOU  WANT  THE  TEXT  PRINTED  OUT?  TYPE  Y  OR  N. 

!  (system  offers  problem 
1  description) 

>y 

t  (student  answers  BIP  query:  y) 

YOU  CAN  ASSIGN  A  LITERAL  VALUE  TO  A  VARIABLE  LIKE  THIS: 

10  T$  «  "TELEPHONE" 

l  (system  qives  hint: 

1  code:  line  10) 

YOU  CAN  ALSO  ASSIGN  THE  VALUE  OF  A  VARIABLE  TO  ANOTHER 

VARIABLE,  LIKE  THIS: 

10  S$  -  T$ 

l  (system  qives  hint: 

!  code:  line  20) 

WRITE  A  PROGRAM  THAT  FIRST  ASSIGNS  THE  VALUE  "ARTICHOKE" 

TO  THE  STRING  VARIABLE  A$. 

!  (system  qives  hint: 

I  desiqn:  line  10) 

THEN  HAVE  IT  ASSIGN  THE  VALUE  OF  A$  TO  ANOTHER  VARIABLE  B$ 

!  (system  qives  hint: 

!  desiqn:  line  20) 

AND  THEN  PRINT  THE  VALUE  OF  B$ . 

J  (svstem  gives  hint: 

!  design:  line  30) 

*10  "artichoke"  a$="artichoke 

l  (student  enters  statement: 
i  line  10) 

A 

SYNTAX  ERROR:  UNMATCHED  QUOTE  MARKS 

l  (system  rejects  statement: 

J  line  10: 

J  syntax_error) 

LINE  NOT  ACCEPTED.  (TYPE  ?  FOR  HELP) 

l  (system  offers  help) 


*10  artichoke 


1  (student  re-enters  statement: 
l  line  10) 


l  (student  requests  hint) 

From  TUTOR :  you're  doing  it  backwards t 

I  (tutor  qives  hint: 
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48 

49 

50 

51 
S7 
53 


56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 


i  design,  line  10) 

Try  typing  the  line  in  this  format : 

10  T  S  =  "TELEPHONE" 

1  (tutor  gives  hint: 

1  code,  line  10) 


10  a$  “"artichoke 


a* 


I  (student  re-enters  statement: 
!  line  10) 


SYNTAX  ERROR:  UNMATCHED  QUOTE  MARKS 

1  (system  rejects  statement: 
1  line  10, 

i  syntax  error) 

LINE  NOT  ACCEPTED.  (TYPE  ?  FOR  HELP)  *” 

!  (system  offers  help) 


*what 


l  (student  enters  command) 


YOU  ARE  IN  TASK  ARTICHOKE. 

DO  YOU  WANT  THE  TEXT  PRINTED  OUT?  TYPE  Y  OR  N. 

!  (system  offers  problem 
!  description) 

>v 


!  (student  requests  system  help) 


[protocol  continues] 


I 
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were  scored  as  desiqn  hints. 

In  response  to  these  hints,  the  student  tried  to  enter  the 
first  line  of  the  proqram  (line  30),  which  was  scored  (line  31) 
as  the  entry  of  line  10  of  BIP's  reference  program.  The  syntax 
error  in  this  line  was  scored  in  BIP's  response  to  the  student's 
input  (line  34).  BIP  then  offered  help  (line  38),  which  the 
student  did  not  request.  Instead,  she  made  another  attempt  at 
the  statement  (line  41),  which  was  aqain  flawed  by  a  syntax 
error.  Before  actually  enter inq  this  line  into  BIP,  however,  she 
requested  a  hint  from  the  tutor  (line  44)  by  pressing  the  touch 
panel. 


In  this  interaction  with  the  student,  the  tutor  provided  two 
hints  of  differinq  specificitv:  a  design  hint  about  the 
assignment  statement  (line  46)  and  a  code  hint  about  the  correct 
form  of  the  assignment  statement  (line  51) .  The  student  then 
tried  to  enter  the  correct  line  (line  55) ,  but  made  another 
syntax  error.  The  student  aqain  rejected  BIP's  offer  of  help 
(line  63) ,  and  entered  the  command  that  will  print  the 
description  of  the  task  (WHAT:  line  65).  As  the  protocol  ended, 
she  asked  to  see  the  task  description  aqain  (line  72) ,  in  which 
she  received  the  design  and  code  hints  that  she  received  in  the 
first  presentation  of  the  problem  description  (lines  8-28) ,  and 
continued  to  work  on  the  problem. 

This  analysis  provided  the  ability  to  describe  a  tutorial 
session  at  an  abstract,  but  still  informative,  level.  The 
student's  problems  were  centered  around  the  assignment  statement 
in  line  10,  having  entered  attempts  at  the  statement  three  times 
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and  having  received  code  and  design  hints  about  the  statement 
from  two  different  sources.  More  importantly,  the  abstract  form 
of  this  analysis  allowed  us  to  evaluate  the  performance  of 
multiole  subjects  on  these  problems,  and  it  is  to  these  kinds  of 
analyses  we  should  now  turn. 

3.3  Taxonomic  Analysis  of  the  BIP  Sessions 

Even  with  the  limited  amount  of  data  we  considered,  the 

taxonomic  analysis  of  these  protocols  revealed  several 

interesting  trends.  Perhaps  the  most  informative  analyses  were 

3 

those  in  which  the  tutorial  sessions  were  divided  in  half,  and 
separate  analyses  were  carried  out  on  each  half.  In  this  way,  we 
could  observe  changes  in  the  performance  of  the  student,  the 
system,  or  the  tutor  as  the  student  gradually  approached  a 
correct  solution. 

3.3.1  Statement  Composition  and  Debugging 

With  the  analysis  of  these  protocols  it  was  possible  to 
address  the  way  students  construct  a  program.  Reca3 1  that  our 
classifications  of  statement  entry  and  reentry  correspond  to  the 
initial  attempts  to  enter  a  syntactically  legal  statement;  any 
modifications  to  a  legal  statement  are  then  scored  as  changes. 

The  mean  frequencies  of  these  classes  of  events  for  our  subjects 
in  the  first  and  second  halves  of  the  protocols  are  shown  in 
Table  4.  These  data  suggest  two  major  processes  in  our  subjects'* 
solution  of  these  problems;  statement  composition  —  in  which  a 
rough  approximation  to  the  proqram  is  constructed  —  and 

3 

On  the  basis  of  the  number  of  lines  of  text  in  the  protocol 
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statement  debugging  —  in  which  the  nearly  correct  components  of 
the  proposed  solution  are  corrected  and  refined. 

These  processes  take  place  neither  strictly  sequentially  nor 
strictlv  simultaneously.  Statement  composition  appears  to  take 
Place  primarily  in  the  first  part  of  the  solution;  our  students 
made  somewhat  more  enters  and  reenters  in  the  first  half  of  the 
protocol.  In  contrast,  statement  debugging  was  concentrated  in 
the  second  half  of  the  solution:  more  than  twice  as  many 
statements  were  changed  in  the  second  half  than  in  the  first. 
Subiects  seem  to  have  first  made  a  general  pass  over  the  problem, 
generating  a  statement  (that  is  not  necessarily  correct)  for  each 
part  of  the  problem,  and  then  gradually  refining  these  statements 
into  their  correct  forms.  This  is  similar  to  Atwood  and 
Jeffries's  (1980)  finding  that  people  approach  complex  software 
design  tasks  by  first  building,  in  a  breadth-first  manner,  an 
approximation  to  a  complete  solution,  and  then  refining  in  a 
depth-first  way  those  components  of  the  solution  that  are  in 
error . 

3.3.2  Students'  Requests  for  Assistance 

Students  in  these  sessions  could  qet  assistance  from  either 
the  human  tutor  (by  pressing  the  touch  panel  and  requesting  a 
hint)  or  by  askinq  BIP  for  help  (the  diagnosis  of  an  incorrect 
statement  could,  if  the  student  wished,  be  followed  bv 
descriptions  of  the  correct  and  incorrect  forms  of  those 
statements) .  Table  5  shows  the  mean  frequencies  of  subjects' 
requests  for  assistance  from  either  the  human  tutor  or  BIP  itself 
during  the  first  and  second  halves  of  the  tutorial  session. 


Table  4: 


Mean  frequencies  of  statement  entries  and  reentries 
vs  statement  changes  in  the  first  and  second  halves 

of  protocols. 


entries 

and  chanqes 

reentries 


first  half  4.8 


3.0 


second  half 


2.4 


6.2 
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Table  5: 

Mean  requests  for  assistance  from  the  human  tutor  and 
the  tutorial  system  in  the  first  and  second 
halves  of  protocols. 


tutor  system 

first  half  1.5  °*5 

second  half  2.1  0*5 
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These  data  show  that  subjects  stronqly  preferred  assistance 
from  the  human  tutor,  particularly  in  the  second  half  of  the 
session,  where,  as  found  in  the  first  analysis,  subjects  were 
debugginq  their  initial  rough  approximation  to  their  program, 
together,  these  analyses  indicate  that  our  subjects  were  willing 
to  accept  BIP's  help  when  they  were  layinq  out  the  initial  plan 
for  the  program,  but  that  they  generally  relied  on  the  human 
tutor  for  assistance  when  debugging  their  programs.  These 
preferences  correspond  to  the  different  kinds  of  help  that  are 
available  from  these  two  sources:  BIP's  system  help  is  good  for 
qeneral  comments  about  the  structure  of  program  statements  and 
the  basic  desiqn  that  the  program  should  take,  while  the  human 
tutor  is  most  valuable  when  the  student  has  entered  a  statement 
that  is  an  incorrect,  but  syntactically  leqal,  piece  of  BASIC 
code.  The  human  tutor  can  identify  the  student's  problem  and 
quide  the  student  toward  a  correct  solution. 

These  data  illustrate  a  further  point  about  our  students' 
use  of  the  system  and  tutor  help  facilities  —  they  were 
extremely  hesitant  to  take  advantage  of  them.  Requests  for 
system  help  were  very  rare  (especially  so  when  it  is  remembered 
that  BIP  offers  this  help  after  every  input  error  made  by  the 
student) ;  requests  for  tutor  hints  were  more  common,  but  still 
infrequent.  The  implications  of  this  findinq  on  the  desiqn  of 
intelligent  tutors  will  be  discussed  later. 

3.3.3  Problem  Difficulty 

Differences  in  the  nature  and  level  of  help  available  from 
the  tutors  and  from  BIP  can  also  be  seen  in  Table  6,  which  shows 
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the  number  of  times  that  students  requested  help  from  either  the 
tutor  or  from  BIP  in  each  of  the  four  tasks.  These  data  indicate 
that  students  preferred  helo  from  the  tutor  at  two  particular 
times:  when  they  were  lust  beqinning  to  use  the  system,  and  when 
they  were  workinq  on  the  hardest  of  the  four  problems. 

Subjects  need  especially  flexible  and  individualized 
assistance  at  both  of  these  times.  When  workinq  on  the  first 
problem,  subjects  needed  help  on  their  use  of  the  BIP  system 
itself,  particularly  which  commands  should  be  used  at  which 
times.  Although  BIP  contains  thorough  descriptions  of  its 
commands  that  could  be  offered  when  a  student  incorrectly  used 
the  command,  it  is  unable  to  infer  from  a  student's  performance 
which  of  these  commands  might  be  appropriate  at  a  given  time; 
therefore,  its  help  facilities  are  of  little  use  at  this  point. 
Further,  since  help  is  available  in  this  svstem  only  after  an 
error  has  been  made,  a  student  may  have  no  choice  but  to  ask  the 
tutor  for  help  in  what  might  be  done  next. 

The  final  and  most  difficult  problem  studied  (TWOS)  requires 
the  use  of  the  BASIC  iteration  statement  (FOR) .  To  solve  this 
problem  correctly,  the  student  must  understand  the  use  of 
iteration  in  the  design  of  the  proqram:  the  ability  to  write  the 
statement  underqoinq  the  iteration  is  assumed,  and  the  skill 
beinq  learned  is  how  to  embed  one  conceptual  part  of  a  proqram 


A 


f 

\ 

'  Table  6: 

Mean  requests  for  assistance  from  the  human  tutor  and 
the  tutorial  system  for  the  four  completed  problems. 


tutor 

system 

ARTICHOKE 

2.0 

O 

• 

o 

SINOP 

2.7 

2.3 

NINOP 

2.0 

2.3 

TWOS 

00 

• 

-4 

.7 

[Note:  these  data  are  ranked  by  the  difficulty  of  the 
problems ;  ARTICHOKE  was  the  easiest  of  the  four;  TWOS  was  the 
Mrdest .  ] 
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within  another.  We  susDect  that  our  students  used  BIP's  helo 
facilities  on  the  simple  problems  because  the  level  of  help  that 
BIP  can  offer  durinq  these  problems  matches  the  level  that  is 
needed  —  advice  on  the  format  of  simple  input/outnut  and 
assiqnment  statements.  However ,  to  effectively  advise  a  student 
on  a  proqram  containinq  iteration,  the  tutor  must  understand  the 
proper  relation  between  the  iteration  statement  and  the  iterated 
statement,  as  well  as  the  student^  (possibly  flawed) 
understanding  of  this  relation.  This  detailed  understanding  of  a 
conceivably  larqe  set  of  proqram  statements  is  beyond  the  scope 
of  a  system  like  BIP. 


3.3.4  Evaluation  of  the  Taxonomic  System 

As  an  experiment  to  determine  whether  such  a  taxonomic 
system  miqht  be  useful  in  the  study  of  tutoring  protocols,  the 
present  analyses  were  quite  promising.  However,  further  research 
involvinq  more  subjects  and  tasks  is  needed  to  refine  this 
technique.  One  direction  in  which  future  research  should  qo  is 
toward  the  generation  of  the  protocol  analysis  by  the  tutorial 
system  itself:  when  a  student  entered  a  statement,  the  system 
could  easily  enter  the  taxonomic  statement  "student  enters 
statement"  into  its  recording  of  the  tutorial  session;  the 


4 

A  stronq  causal  statement  regard inq  why  students  stop 
reauesting  hints  on  hard  problems  cannot  be  made  from  these  data, 
since  the  difficulty  of  the  problems  received  by  a  student  is 
confounded  with  number  of  problems  that  a  student  has  solved. 
While  we  believe  that  the  "canned"  hints  like  those  offered  by 
BIP  are  initially  useful,  but  become  less  useful  as  the  problems 
become  more  complex,  it  may  also  be  that  canned  hints  are  never 
reallv  useful  to  the  students,  but  that  the  students  must  work 
with  BIP  for  some  time  before  they  realize  this. 
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system's  acceptance  or  rejection  and  diagnosis  of  that  statement 
could  similarly  be  noted.  However,  some  decisions  cannot  be  made 
as  easily  as  these  simple  notations  of  syntax  errors:  if  a 
student  entered  a  poorly  constructed  IF  statement  during  the 
composition  of  a  proqram  that  contained  two  IF  statements,  it 
would  probably  difficult  to  determine  which  of  the  two  the 
student  was  tryinq  to  write.  At  this  point  the  taxonomic 
analysis  of  protocols  would  begin  to  change  from  a  simple 
transcription  of  the  "surface  structure"  of  a  proqram  into  a  more 
conceptual  analysis  of  the  student's  plan  that  underlies  the 
generated  program  —  exactly  the  kind  of  analysis  that  is  needed 
in  an  "intelligent"  tutor.  It  is  unclear  at  this  time  whether  a 
taxonomic  system  like  that  used  here  might  also  serve  as  an 
abstract  descriptive  lanquage  that  could  be  exploited  by  the  plan 
identification  component  of  an  intelligent  tutoring  system  (cf. 
Miller  &  Goldstein,  1977),  but  this  possibility  may  be  worth 
exploring. 

3.4  Detailed  Analysis  of  a  BIP  Session 

Although  much  can  be  learned  from  the  taxonomic  analyses 
described  above,  a  more  detailed  understanding  of  the  tutorial 
interaction  still  requires  analysis  of  the  oriqinal  protocols. 

One  of  our  subjects  —  referred  to  as  DD  —  found  the  programming 
tasks  particularly  difficult,  and  was  the  focus  of  such  an 
analysis.  Her  protocol  offers  offers  a  particularly  rich  source 
of  data  on  the  interaction  between  a  tutor  and  a  student. 

DD  correctly  completed  the  GREENFLAG,  ARTICHOKE,  SINOP,  and 
NINOP  tasks,  and,  in  doing  so,  demonstrated  knowledge  of  the 
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following  asoects  of  programming  in  BASIC: 

-Numerical  and  string  constants 

-Assignment  of  a  string  or  numerical  constant  to  a 
variable  (i.e.,  A=5  or  A=" ARTICHOKE") 

-Assignment  of  a  variable's  value  to  another  variable 
(i.e.,  A=B) 

-INPUT  and  PRINT  statements 


DD  then  moved  on  to  the  TWOS  task,  which  is  described  by  BIP 
as  follows: 

Write  a  program  that  counts  by  twos,  up  to  a  number  given 
by  the  user.  For  example,  if  they  gave  8,  your  program 
would  print: 

2 

4 

6 

8 

Use  a  FOR  .  .  NEXT  loop  for  this  problem. 


This  oroblem  can  be  solved  by  the  following  BASIC  program: 

1  REM  N  IS:  THE  NUMBER  TO  COUNT  UP  TO 
10  PRINT  "HOW  HIGH  SHOULD  I  COUNT?" 

20  INPUT  N 

30  FOR  I  =  2  TO  N  STEP  2 
40  PRINT  I 
SO  NEXT  I 
99  END 


The  new  skill  introduced  in  this  task  is  the  use  of  the  FOR 
statement  for  the  construction  of  iterative  loops.  This  task  was 
selected  bv  BlP's  curriculum  information  network  because  DD  had 
successfullv  completed  tasks  that  required  using  inout/output 
statements,  the  only  other  statements  needed  to  solve  this 
problem. 


DD  worked  on  this  task  for  almost  two  hours,  but  never  wrote 


a  completelv  successful  program.  Her  nearly  complete  program 
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failed  in  a  way  that  caused  a  fatal  error  condition  in  BIP  which 

5 

terminated  the  session.  The  analysis  of  DD's  protocol  revealed 
some  serious  confusions  about  BASIC  and  computer  programming  in 
general: 

*  She  was  confused  about  variable  assignment.  Her  protocol 
suggests  that  she  initially  thought  that  the  INPUT  statement 
should  be  used  to  assign  values  to  variables  (i.e.,  "INPUT  X  8" 
means  "set  X  to  R") ,  and  that  she  did  not  really  understand  that 
using  an  INPUT  statement  in  her  program  would  allow  her  to  enter 
a  value  when  she  ran  the  program  at  a  later  time.  It  is 
important  to  note  that  she  was  confused  in  this  way  even  though 
she  had  successfully  completed  several  other  BIP  tasks  that 
required  her  to  use  INPUT  and  assignment  statements. 

*  DD's  handling  of  the  FOR  statement  required  by  this 
program  suggested  major  misunderstandings  at  several  different 
levels.  She  tried  several  different  (and  illegal)  forms  of  the 
FOR  statement  (such  as  "FOR  X  =  2  -  8”),  had  problems  determining 
the  exact  relation  between  the  FOR  and  NEXT  statements,  and  was 
unsure  of  the  proper  way  to  incorporate  the  STEP  information  into 
the  FOR  statement.  Beyond  the  syntax  of  the  FOR  statement,  she 
had  trouble  grasping  two  ideas  that  are  concerned  with 
higher-level  aspects  of  programming:  (a)  embedding  one 
conceptual  group  of  statements  (the  statements  to  be  repeatedly 

- 

DD"s  program  consisted  of  the  statements  shown  in  the  sample 
program,  but  her  specification  of  the  upper  bound  and  the  index 
variable  of  the  FOR  was  incorrect.  It  is  unclear  how  much 
additional  work  by  PD  would  have  been  required  to  correct  this 
error . 
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executed)  within  another  group  of  statements  (the  FOR  and  NEXT 
statements  that  controlled  the  iteration) ,  and  (b)  using  the 
FOR"*s  index  variable  for  some  purpose  in  the  statements  within 
the  iterative  looo.  Although  she  finally  wrote  a  program  that 
had  the  proper  collection  of  statements  in  the  right  order  — 
INPUT,  FOR,  PRINT,  NEXT,  and  END  —  she  confused  the  uses  and 
functions  of  the  FOR's  index  variable  and  the  number,  entered 
through  the  INPUT  statement,  that  controlled  the  upper  bound  of 
the  loop. 

*  She  never  had  a  good  understanding  of  the  overall  design 
of  the  program.  The  use  of  a  variable  to  soecifv  the  upper  bound 
of  the  loop  was  the  last  part  of  the  program  to  be  written; 
throughout  the  maiority  of  the  session,  she  wrote  FOR  statements 
with  exolicit  upper  bounds,  either  B  or  10.  She  used  the  MORE 
command  —  the  command  that  tells  RIP  that  the  student  has 
finished  working  on  one  problem  and  wants  to  go  on  to  another  — 
several  times  with  programs  that  contained  such  an  explicit  upper 
bound,  indicating  that  she  thought  she  had  a  correct  Program  even 
with  that  incorrect  construction. 

Not  all  these  problems  can  be  blamed  on  DD  herself.  Rather, 
some  of  her  Problems  can  be  attributed  to  various  aspects  of  the 
human  tutor,  the  modified  version  of  rtp,  and  BASIC. 

3.4.1  Problems  with  BASIC 

WASTC  is  an  inoreae  ing1 v  common  first  computer  annuag°,  tut 
it  has  man-'  disadvantageous  aspects  for  bea  inning  programmers. 
m  ip-'  or  ~  .  ire  related  to  notions  of  structured  nroqr  ammin  i . 
t»r  •••'!  r  -  f  n  \  B  t  <■’  *  s  extreme!'/  limited  ,-ont'  it 
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structures  can  be  very  difficult  to  read  and  understand.  In 
addition,  the  restricted  length  of  BASIC  variable  names  (which 
makes  the  use  of  meaningful  mnemonics  difficult)  and  the  strict 
format  of  statements  (a  line  must  contain  exactly  one  complete 
statement)  can  also  be  expected  to  complicate  the  programming 
task . 

3.4.2  Problems  with  BIP 

There  are  two  aspects  of  BIP  that  should  be  improved  in 
future  tutorial  systems: 

1:  Limited  display.  'J’he  student's  interaction  with  BIP  must 
take  place  within  the  twenty-four  lines  of  a  computer  terminal's 
screen.  This  greatly  restricts  the  student's  ability  to  work 
simultaneously  with  a  tutor,  his  program,  and  BIP.  Twenty-four 
lines  are  not  enough  to  allow  a  student  to  keep  track  of  the 
current  state  of  his  program,  the  output  produced  by  that 
oroqram,  recent  changes  made  to  the  program,  interactions  with 
BIP,  and  hints  obtained  from  several  possible  sources.  DD  often 
entered  proqram  lines  that  were  already  present  (at  one  point, 
CD's  TWOS  proqram  had  four  END  statements) ,  entered  new  lines 
with  the  same  statement  number  as  already-entered  statements 
(thereby  erasing  the  old  and  possibly  correct  statements) ,  and 
ran  the  proqram  with  certain  statements  missing.  A  system  that 
used  multiple  "windows"  to  display  the  various  components  of  the 
tutorial  interaction  would  avoid  many  of  these  problems.  Such 
terminals  were  not  available  when  BIP  was  built,  but  future  tutor 
projects  should  take  advantage  of  these  systems. 

2:  Rigid  input  format.  BIP  offers  no  provisions  for 


44 


correcting  spellinq  or  typing  errors.  Hence,  when  DD  typed  "RU 
N"  instead  of  "RUN",  BIP  could  do  nothing  more  than  identify  it 
as  an  illegal  command,  if  tutorial  systems  for  beginning 
programming  require  the  students  to  enter  their  programs  on  a 
kevboard  (rather  than  choosing  command  names  and  arguments  from  a 
menu,  for  instance),  automatic  spelling  correction  miqht  be 
useful. 

3.4.3  Problems  with  the  Tutor 

Certain  problems  in  DD's  performance  can  also  be  traced  to 
the  tutor.  In  general,  our  tutors  —  skilled  computer 
programmers  who  generally  did  not  have  substantial  teaching 
experience  —  did  quite  well.  When  a  hint  was  requested,  they 
were  consistently  able  to  identify  the  student's  problem  and 
offer  a  relevant  hint.  Whether  or  not  that  hint  was  the  "right" 
or  the  "best"  hint  is  beyond  the  scope  of  this  research.  For 
now,  the  best  evidence  for  the  appropriateness  of  our  tutors' 
hints  is  that,  after  receiving  hints,  students  always  worked  on 
their  problem  before  requesting  another  hint;  on  no  occasion  did 
a  student  indicate,  by  asking  for  two  consecutive  hints,  "I 
understand  that  —  my  problem  is...". 

Our  tutors'  problems  stem  primarily  from  the  fact  that  the 
tutorial  domain  is  very  open-ended,  and  that  instantaneouslv 
generating  high-quality  hints  about  any  aspect  of  a  programming 
problem  and  a  student's  solution  to  that  problem  is  not  easy. 

Some  of  these  problems  were  mechanical  and  avoidable  — 
occasionally,  a  tutor  would  start  to  type  a  hint,  change  his 
mind,  and  erase  what  he  had  written,  one  character  at  a  time. 
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All  of  this  was  visible  to  the  student  in  these  experiments;  a 
minor  modification  would  allow  the  tutor  to  compose  his  hint 
carefully  and  send  it  to  the  student  only  when  he  was  satisfied 
with  its  form. 

More  relevant  to  this  discussion,  however,  are  the  occasions 
when  the  tutors"  hints  were  inconsistent  or  ambiguous.  For 
instance,  when  DD  was  trying  to  construct  a  legal  FOR  statement, 
the  following  interaction  took  place; 

a:  DD  requested  a  hint,  with  her  program  in  the  following 
state : 

10  FOR  X  =  2  TO  8  STEP  2 
20  PRINT  X 
30  END 

b;  The  tutor  said  "You  have  a  FOR  on  line  10,  but  nowhere  do 
you  have  a  NEXT." 

c;  DD  tried  to  change  line  10  to: 

10  FOR  X  =  2  TO  8  NEXT  STEP  2 

When  RIP  rejected  this  illegal  statement,  DD  asked  for  another 
hint . 


d:  The  tutor  said  "I"m  sorry,  you  misunderstood  my  last 
hint.  You  need  a  NEXT  statement  on  a  different  line  AFTER  the 
PRINT  statement . . . . " 

The  source  of  this  error  lies  in  the  differing  knowledge 
structures  possessed  by  the  student  and  tutor  that  guide  their 
individual  comprehension  of  the  sentence.  While  it  is  clear  to 
the  tutor  that  NEXT  belongs  on  a  separate  line  —  because  of  his 
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prior  experience  with  BASIC  —  it  is  not  so  clear  to  DD,  and  the 
observed  misinterpretation  occurs.  This  incident  might  have  been 
avoided  if  the  tutor  had  said  in  his  first  hint,  "...nowhere  do 
you  have  a  NEXT  statement. " 

A  related  problem  with  the  different  knowledge  structures 
possessed  by  student  and  tutor  was  illustrated  by  another  part  of 
DD's  session.  DD  had  requested  a  hint  when  she  was  having 
problems  writing  a  correct  FOR  statement;  the  tutor  offered  the 
following  hint: 

If  we  want  to  set  A  to  go  from  1  to  10  and  print  A  we 

would  write  the  following  program: 

10  FOR  A-l  TO  10 

20  PRINT  A 

30  NEXT  A 

This  is  similar  to  your  task,  except  that 

(a)  you  need  to  go  from  2  to  whatever  number  the  user 

typed. 

(b)  you  want  to  count  by  twos. 

This  is  done  by  saying: 

FOR  variable  =  start  TO  finish  STEP  2. 

DD  responded  to  this  hint  by  entering  the  statement: 

FOR  VARI ABLFi=START  TO  FINISH 

The  problem  exposed  here  is  that  the  tutor  is  using  a  particular 
notational  system  common  to  experienced  computer  programmers: 
statements  are  described  bv  presenting  their  required  keywords  in 
capital  letters  (such  as  FOR  and  TO) ,  and  presenting  variables 
that  must  be  specified  by  the  programmer  in  lower  case  terms  that 
describe  their  function  (e.g.,  variable,  start,  and  finish) .  DD 
does  not  possess  this  structure,  so  she  interprets  the  tutor's 
hint  too  literally.  Mismatches  of  this  nature  were  common  in  our 
protocols;  at  one  point,  the  tutor  was  explaining  the  use  of  the 


STEP  keyword  in  the  FOR  statement: 


To  tell  the  loop  what  to  step  by  use  the  following  form: 

10  FOR  something  =  something  TO  something  STEP  NUMBER 

where  NUMBER  is  the  number  you  want  to  step  by. 

To  count  by  5's  you  would  say  ...  STEP  5. 


DD  then  entered: 
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10  FOR  X  =  2  TO  8 
20  STEP  2 

The  mismatch  here  lies  in  the  tutor's  use  of  "..."  to  stand 

for  the  part  of  the  FOR  statement  she  had  already  written.  In 

addition,  the  problem  exists  because  of  BASIC's  requirement  that 

the  STEP  information  be  on  the  same  line  as  the  FOR  statement;  a 

less  rigidly  formatted  language  would  not  have  nosed  this 
7 

problem. 


These  problems  illustrate  the  advantages  and  disadvantages 
of  "canned"  hint  systems  and  intelligent  tutoring  systems.  While 
it  is  impossible  to  anticipate  every  occasion  for  which  a  canned 
hint  should  be  constructed,  hints  can  be  very  carefully  worded 
for  those  circumstances  that  can  be  anticipated.  In  contrast, 
while  human  tutors  possess  the  problem-solvinq  power  systems  such 
as  BIP  lack,  the  task  of  diagnosing  a  student's  problem,  deriving 


6 

Note  DD's  failure  to  use  a  variable  for  the  loop's  upper 
bound . 

7 

On  the  other  hand,  such  a  language  would  not  have  revealed 
this  problem  in  DD's  conceptualization  of  the  task,  and  DD  would 
probably  have  written  her  future  FOR  loops  with  the  STEP 
information  on  a  separate  line. 


i 
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a  sound  hint  for  that  problem,  and  converting  that  hint  to  an 
unambiguous  piece  of  natural  language  (all  while  being  under  the 
time  pressure  of  trying  to  generate  this  hint  as  quickly  as 
possible)  is  extremely  complex  for  even  human  tutors. 

3.4.4  Problems  with  the  "Hint  Button"  Approach 

We  originally  proposed  to  study  tutoring  systems  with  "hint 
buttons"  as  a  way  of  isolating  one  part  of  the  intelligent 
tutoring  domain.  By  providing  hints  only  upon  request,  we  could 
bypass  the  complex  issue  of  identifying  the  circumstances  under 
which  the  tutorinq  system  should  interrupt  the  student  with  a 
hint,  and  concentrate  our  efforts  on  techniques  for  qeneratinq 
these  hints.  These  techniques  are  described  in  the  discussion  of 
TURTLE  (Section  4).  However,  with  the  present  experiments,  we 
can  address  the  question  of  whether  successful  tutors  might  be 
built  by  takinq  this  shortcut,  and,  like  the  modified  BIP  system 
and  TURTLE,  giving  hints  only  when  a  student  requested  them. 

These  experiments  suggest  that  such  a  shortcut  would  not  be 
reasonable. 

A  surprising  finding  of  these  experiments  was  that  our 
subjects  were  hesitant  to  request  hints  from  the  tutor;  requests 
averaged  onlv  about  four  per  session  (Table  6) .  The  fact  that, 
in  these  experiments,  the  students  had  to  ask  another  person  for 
help  does  not  seem  to  be  part  of  the  problem.  The  students  were 
also  unlikelv  to  take  advantage  of  BIP's  help  facilities,  and,  in 
some  informal  experiments,  there  was  a  similar  hesitancy  to 
request  hints  on  the  part  of  students  who  worked  with  TURTLE, 
whose  hint  facility  was  completely  automated. 


Our  experiences  with  both  tutorial  systems  sugqest  that  one 


of  the  commonlv  cited  educational  advantages  of  programming  — 
that  writing  a  computer  proqram  is  often  like  solving  a  nuzzle  or 
even  playing  a  game  —  works  against  the  idea  of  a  "hint  button". 
Subjects  seemed  to  treat  the  use  of  the  hint  button  as  cheating, 
or  something  that  should  be  done  only  as  a  method  of  last  resort. 


It  could  be  argued  that  our  subjects  were  simply  not 
motivated  strongly  enough  to  request  hints,  and  that  more 
encouragement  would  lead  to  more  requests  for  hints.  However, 
this  would  ignore  the  long-range  problem  of  the  design  of  an 
"intelligent",  hint-giving,  tutor.  One  of  the  reasons  our  tutors 
could  successfully  identify  the  student's  problem  when  a  hint  was 
requested  was  because  students  requested  hints  so  infrequently. 
When  DD  entered 


10  FOR  X  =  2  TO  8 
20  STEP  2 
20  STEP  2 
20  X  =  STEP  2 
STEP  2 
20  STEP  2 
20  STEP  2 


flagged  as  illegal  by  BIP 
flagged  as  illegal  by  BIP 
flagged  as  illegal  by  BIP 
flagged  as  illegal  by  BIP 
flagqed  as  illegal  by  BIP 
flagged  as  illegal  by  BIP 


before  finally  asking  for  a  hint,  it  was  clear  she  was  havinq 
trouble  with  the  syntax  of  the  FOR/STEP  statement.  Without  this 
reoetition,  the  tutor  miqht  not  have  been  able  to  identify  her 
oroblem.  Suppose  DD  had  entered  line  10,  as  in  the  above 
example,  and,  after  realizing  that  she  did  not  know  how  to 
soecify  the  STEP  statement,  asked  for  a  hint.  Working  only  with 
the  sinqle  statement  thus  far  entered  by  the  student,  the  tutor 
could  offer  hints  referring  to  the  absence  of  of  the  STEP 
component  of  the  FOR  statement,  the  missing  NEXT  statement,  or 
the  fact  that  the  FOR  uses  an  explicit  upper  bound  rather  than  a 
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variable.  Onlv  one  of  these  alternatives  would  correspond  to  the 
student's  real  problem,  and  the  tutor  would  need  additional 
evidence  to  select  the  right  one.  With  frequent  hints  and  no 
supplementary  information  from  the  student  to  help  identify  his 
problem,  effective  tutoring  would  be  difficult. 

The  study  of  these  protocols  leads  to  two  conclusions. 

First,  interruption  by  the  tutorial  system  to  give  certain  hints 
would  not  be  bevond  the  capability  of  current  "intelligent 
tutoring  systems"  technology.  Many  of  the  hints  students 
requested  were  preceded  by  repeated  syntax  errors  like  those 
shown  above.  While  the  comprehension  and  diagnosis  of  a  computer 
program's  desiqn  is  considerably  more  difficult  than  identifying 
syntax  errors,  a  tutorial  system  capable  of  diagnosinq  repeated 
syntax  errors  and  interrupting  the  student  with  a  hint  about  the 
statement's  nrooer  form  would  not  be  difficult.  Further,  a 
detailed  theory  of  how  people  learn  to  program  is  not  really 
needed  in  order  to  suggest  that  a  student  who  makes  several 
consecutive  identical  errors  should  be  interrupted  with 
corrective  advice  (although  such  a  theory  would  be  critical  to 
generating  the  best  possible  hint) . 

Second,  the  communication  between  a  student  and  a  tutorial 
svstem  should  be  more  flexible  than  allowing  the  student,  in 
effect,  to  say  nothing  more  than  "I  need  help!".  In  most  cases, 
the  diagnosis  of  a  student's  problem  will  be  facilitated  bv 
allowing  the  student  to  describe  why  he  is  requesting  a  hint. 
Natural  language  would  ultimately  be  useful  for  this  purpose; 
although  the  difficulties  of  building  natural  language  systems  in 
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unconstrained  domains  are  well-known,  a  tutorial  dialogue  in  a 
particular  domain  might  provide  sufficient  constraint  to  make 
such  a  system  possible  (cf.  Brown  &  Burton,  1975;  Stevens  & 
Collins,  1977).  Alternatively,  the  tutorial  svstem  might  offer  a 
menu  of  problems  the  student  could  be  having,  and  allow  the 
student  to  identify  the  relevant  one.  A  simple  system  miqht  have 
a  predefined  menu  that  was  meant  to  cover  all  the  problems  a 
student  miqht  have,  while  a  more  advanced  svstem  might  infer  a 
smaller  set  of  possible  problems  from  the  tutorial  context.  If  a 
student  workinq  on  the  TWOS  problem  entered  "FOR  I  =  2  TO  8"  and 
requested  a  hint,  the  system  miqht  offer  the  menu; 

Do  you  want  help  on: 

1:  the  STEP  component  of  the  FOR  statement 

2:  the  NEXT  statement 

3:  the  use  of  variables  in  FOR  statements 

4:  something  else 

5:  nothing  (you  don't  want  any  help) 

These  topics  for  possible  hints  would  be  identified  by 
comoarinq  the  correct  statement  to  the  statement  entered  by  the 
student  and  offer inq  help  about  each  of  the  mismatches  between 
the  statements  (e.q.,  the  absence  of  the  STEP  component  in  the 
FOR  statement,  the  unentered  NEXT  statement,  and  the  improper  use 
of  a  numerical  constant  as  the  upper  bound  of  the  loop)  or  some 
other  problem  not  included  in  this  list,  as  well  as  the  option  of 
rejectinq  help. 

3.5  Advantages  of  a  Human  Tutor 

What  are  the  properties  of  a  human  tutor  that  make  his 
suogestions  more  valuable  to  a  student  than  the  help  offered  by 
BIP?  The  assistance  available  from  the  tutor  and  from  BIP 
differed  in  two  ways: 


*  Flexibility;  Unlike  BIP's  help  messages,  which  are  tied  to 


specific  command  or  statement  errors,  a  human  tutor  can,  at  anv 
time,  offer  help  on  any  part  of  the  tutorial  interaction.  In 
addition,  a  tutor  can  properly  interpret  a  very  larqe  number  of 
ways  of  solving  a  particular  problem,  whereas  BIP's  diagnosis  of 
a  student's  program  assumes  a  particular  approach  to  the  problem. 

*  Specificity;  Although  BIP  can  describe  in  general  terms 

the  statements  that  comprise  a  program  fr.e.,  "...  vou  need  an 

IF  statement  and  two  PRINT  statements..."),  identify  illegal 

statements,  and  offer  assistance  on  the  proper  forms  of  these 

statements  when  errors  occur,  it  can  offer  very  little  help  once 

the  student  has  entered  a  statement  that  is  syntactically  leqal, 

but  incorrect  in  the  context  of  the  current  problem.  In 

contrast,  a  human  tutor  can  evaluate  a  legal  statement  and 

determine  that  some  subpart  of  the  statement  —  perhaps  the 

variable  that  specifies  the  upper  bound  of  a  FOR  statement  —  is 

at  fault.  Recall  that  the  original  version  of  BIP  does  have  a 

"hint"  facility,  throuqh  which  students  can  receive  hints  by 

issuing  the  BIP  command  "HINT",  but  since  these  hints  are 

predefined  in  much  the  same  way  as  the  help  BIP  offers  after 

statement  errors,  this  facility  can  offer  neither  flexibility  nor 

soecificitv.  For  instance,  BIP's  hint  for  TWOS  is; 

The  -FOR-  statement  can  make  the  loop  count  by  twos 
automatically.  Look  for  an  explanation  of  the  'step' 
part  of  the  statement. 

Although  this  hint  may  sometimes  be  useful,  it  would  not  have 
been  much  help  to  DO,  who  knew  that  she  had  to  specify  STEP 
information,  but  did  not  know  the  right  way  to  do  this. 


■  A 
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In  human  tutors  these  properties  come  from  knowledge  of  the 
problem  at  hand,  general  programming  and  problem  solving 
techniques,  and  a  variety  of  educational  and  tutorial  strategies. 
The  second  portion  of  this  report  describes  a  system  that, 
through  its  partial  representation  of  these  kinds  of  knowledge, 
can  give  a  student  more  flexible  and  specific  tutorial 
assistance.  While  this  work  is  only  a  beginning,  it  brings  us  a 
step  closer  to  more  "intelligent"  computer-based  tutoring. 
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Section  4 

Program  Understanding  and  Synthesis  in  LOGO:  The  TURTLE  Tutor 

TURTLE  is  a  tutorial  system  that  provides  "intelligent" 
assistance  for  a  student  solvinq  a  set  of  tasks  in  LOGO 
orogramminq:  this  assistance  is  intended  to  be  approDriate  to  the 
student's  plan  for  solvinq  the  task  and  specific  to  the  problem 
facinq  the  student  when  help  is  requested.  TURTLE  uses  analysis 
bv  synthesis  techniques  to  interpret  the  student's  program,  to 
qenerate  hints  upon  request  by  the  student,  and  to  assist  the 
user  in  correcting  flawed  programs. 

LOGO  proqrams  draw  figures  on  the  screen  of  a  graphics 

terminal  by  simulating  the  movement  of  a  "turtle".  The  turtle 

can  draw  a  line  on  the  screen  corresponding  to  its  movement.  At 

the  beqinninq  of  a  session,  the  turtle  is  in  the  center  of  the 

screen,  facing  north;  its  position  can  be  chanqed  with  the 
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following  primitives  : 

-DRAW  X:  move  forward  X  units,  drawing  a  line. 

-MOVE  X:  move  forward  X  units  without  drawinq  a  line. 

-TURN  X:  turn  the  turtle  X  degrees  clockwise. 

A  student's  proqram  can  be  made  up  of  these  primitives  and 
references  to  user  defined  functions.  The  current  implementation 
of  TURTLE  does  not  support  recursion,  iteration,  variable 
assignment,  or  subroutine  calls  more  than  one  level  deep. 

- 

Note  that  these  Drimitives  differ  sliqhtly  from  those 
described  by  Paoert  (1980)  ,  and  in  fact  correspond  to  those 
defined  in  OGOL  (Section  2). 


Programs  that  meet  these  constraints  can  draw  simple  pictures 
like  TRIANGLE,  TREE,  WELL,  and  NAPOLEON  (Figure  2),  the  four 
tasks  that  form  the  current  version  of  TURTLE. 


A  student  workinq  with  TURTLE  is  qiven  the  task  of  writing  a 
LOGO  program  that  will  draw  one  of  these  figures.  This  program 
may  simply  be  a  series  of  LOGO  statements  —  an  open  coded 
oroaram  —  or  it  may  decompose  the  fiqure  into  a  number  of 
subfigures,  each  of  which  is  drawn  by  a  function  defined  by  the 
student.  For  instance,  the  WELL  (see  Figure  2)  can  be  decomposed 
into  the  triangle  at  the  top  of  the  figure,  the  square  at  the 
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bottom,  and  the  line  that  connects  the  triangle  and  the  square. 

In  whatever  way  the  program  is  written,  TURTLE'S  task  is  to 
identify  the  olan  used  by  the  student  to  construct  this  program 
and  to  use  this  Dlan  to  help  the  student  locate  and  correct 
errors  in  the  oroqram. 

4.1  Sample  TURTLE  Sessions  for  the  TRIANGLE  and  WELL  Tasks. 

When  a  student  beqins  a  session  with  TURTLE,  he  is  presented 
with  a  general  discussion  of  computers  and  the  LOGO  language. 
During  this  discussion,  he  can  experiment  with  the  various  LOGO 
primitives  by  enterinq  statements  and  observing  their  results. 

The  student  then  solves  the  TRIANGLE  and  TREE  tasks.  Next,  the 
concent  of  program  decomposition  is  introduced,  and  the  student 
is  encouraged  to  use  decomposition  to  solve  WELL  and  NAPOLEON. 

- 

Other  decompositions  are  possible,  of  course;  TURTLE'S 
analysis  of  problems  with  several  different  decompositions  is 
described  in  Section  4.2.5. 


:igure  2  Sample  TRIANGLE 


4.1.1  TRIANGLE:  Interpreting  Open  Coded  Solutions 

The  first  task  presented  to  students  by  TURTLE  was  TRIANGLE, 
in  which  the  student  was  to  write  an  open  coded  proqram  to  draw 
an  equilateral  trianqle  (see  Figure  2;  note  that  the  triangle  is 
"pointing"  upward) .  The  following  is  a  trace  of  a  hypothetical 
subject's  solution  of  this  problem;  TURTLE'S  output  is  in 
boldface.  This  section  presents  only  a  sample  interaction  with 
TURTLE  —  a  detailed  description  of  how  TURTLE  operates  will 
follow. 

TURTLE  first  displays  the  correct  solution  of  the  triangle 
task  on  the  graphics  terminal;  this  figure  can  be  drawn  by  a 
program  that  starts  at  vertex  A  (Figure  2)  and  proceeds  as 
follows : 

TURN  30 
DRAW  200 
TURN  120 
DRAW  200 
TURN  120 
DRAW  200 


The  student  is  then  asked  to  define  TRIANGLE: 


TRIANGLE  is  not  yet  defined.  Please  enter  the  steps  in 
TRIANGLE.  (Type  QUIT  to  end  the  function  definition, 
RESTART  to  start  over,  and  DELETE  to  delete  the  last 
step. ) 

First  step?  DRAW  200 
Next  step?  TURN  60 
Next  step?  DRAW  200 
Next  step?  QUIT 
OK,  TRIANGLE  is  defined! 


There  are  several  errors  in  the  student's  proposed  program. 


Onlv  two  of  the  three  sides  of  the  triangle  are  drawn,  and  the 
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rotation  between  the  first  two  steps  should  be  TURN  120,  rather 
than  TURN  60.  Despite  the  presence  of  these  errors,  TURTLE 
discovers  three  paths  through  the  triangle  that  provide 
relatively  good  matches  to  the  student's  program,  and  asks  the 
student  which  of  these  corresponds  to  his  path  through  the 
figure: 

Did  you  to  intend  to  follow  path  C  A  B  C?  NO 

Did  you  to  intend  to  follow  path  B  C  A  B?  NO 

Did  you  to  intend  to  follow  path  A  B  C  A?  YES 

Having  identified  the  path  used  by  the  student,  TURTLE 

generates  the  following  analysis  of  the  student's  program,  which, 
if  desired,  can  be  presented  to  the  student: 

Analysis  of  your  program: 


Student 

Code 

Correct 

Code 

Commentary 

DRAW  200 

DRAW  200 

Correct  mainstep  to  draw 
line  segment  AB 

TURN  60 

TURN  120 

Incorrect  interface  to  turn 
through  angle  ABC 

DRAW  200 

DRAW  200 

Correct  mainstep  to  draw 
line  segment  BC 

TURN  120 

Missing  interface  to  turn 

through  angle  BCA 

DRAW  200 

Missing  mainstep  to  draw 
line  segment  CA 

TURTLE  then  displavs  the  figures  corresponding  to  desired 
solution  and  the  student's  current  solution  on  the  graphics 
terminal,  and  begins  the  debugging  process: 


Disregarding  the  orientation  for  now,  what  bugs,  if 
any,  do  you  see  in  the  function  TRIANGLE? 

1  -  NO  BUGS 

2  -  MISSING  STEPS 

3  -  INCORRECT  STEPS 

4  -  EXTRA  STEPS 

>  2 


The  student  says  that  there  are  missinq  steps;  TURTLE 
agrees,  and  offers  further  information: 

Yes,  there  are  MISSING  STEPS  in  the  function  TRIANGLE. 

You  did  not  mention  the  following  bugs:  The  function 

TRIANGLE  has  INCORRECT  STEPS. 

At  this  point,  TURTLE  shows  the  student  his  program  and 
allows  him  to  make  modifications  or  to  request  a  hint;  the 
student  requests  a  hint. 


Here  are  the  steps  you  are  editing: 

DRAW  200 
TURN  60 
DRAW  200 

What  would  you  like  to  do? 

1  -  SHOW  FUNCTION 

2  -  BUGS  REPORT 

3  -  DELETE  STEP 

4  -  CHANGE  STEP 

5  -  INSERT  STEP 

6  -  RECEIVE  HINT 

7  -  FINISH  EDITING 

>  6 

[RECEIVE  HINT] 


Here,  TURTLE  offers  a  qeneral  hint  about  the  missinq  step  — 
You  are  missing  the  line  segment  from  C  to  A  —  after  which  the 
student  asks  for  another  hint.  This  hint  is  more  specific: 


You  are  missinq  the  step  DRAW  200  to  draw  the  line 
segment  from  C  to  A. 
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At  this  point,  the  student  has  been  returned  to  the  editing 
menu,  asks  to  insert  a  step,  and  enters  DRAW  200  at  the  end  of 
his  program,  which  will  draw  the  third  side  of  the  triangle.  He 
then  requests  another  hint,  and  receives  help  on  one  of  the 
missing  TURNs: 

You  are  missing  the  rotation  to  turn  through  angle  BCA. 

When  this  hint  is  displayed,  the  line  segments  that  make  ud 

anqle  BCA  are  highlighted  in  the  displays  of  the  correct  drawinq 

and  of  the  results  of  the  student's  current  proqram.  The  student 

then  asks  to  enter  the  step  TURN  60  between  the  latter  two  DRAW 

200's.  However,  TURTLE  rejects  this  addition: 

That  step  does  not  need  to  be  inserted  anywhere.  A 
TURN  does  need  to  be  inserted,  but  the  value  60  is 
incorrect. 

At  this  point,  the  student  determines  that  a  120  degree  turn 
is  needed,  and  enters  a  TURN  120;  the  result  of  this  revised 
proqram  is  then  displayed.  The  student  now  asks  for  another 
hint;  TURTLE  replies 

The  rotation  to  turn  through  angle  ABC  is  incorrect. 

and  hiqhlights  the  rotations  in  both  the  correct  drawing  and  the 
student's  current  drawinq.  The  student  continues  by  requesting 
another  hint,  and  is  told: 

The  rotation  to  turn  through  angle  ABC  should  be 
TURN  120  instead  of  TURN  60. 

Here,  the  student  uses  the  CHANGE  STEP  option  in  the  editinq 
menu  to  change  the  TURN  to  120  degrees,  and  then  selects  FINISH 
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EDITING.  The  function  is  now  correct,  and  TURTLE  reports:  The 
function  TRIANGLE  has  no  bugs! 

Although  the  function  itself  is  correct,  the  figure's 
orientation  is  not:  the  trianqle  "points"  to  the  right,  rather 
than  upward.  TURTLE  now  asks  the  student  to  specify  interface 
steDS  that  will  correct  this  orientation  error: 

You  provided  no  setup  for  the  TRIANGLE-TASK.  What 
bugs,  if  any,  do  you  see  in  the  orientation  of  TRIANGLE? 

The  student  says  that  there  are  missing  steps,  and  TURTLE 
aqrees : 

Yes,  there  are  MISSING  STEPS  in  the  orientation  of 
TRIANGLE.  No  setup  interface  for  TRIANGLE  has  been 
provided.  What  would  you  like  to  do? 

Here,  the  student  asks  for  a  hint  regarding  the  interface, 
and,  after  receiving  it  — 

Another  TURN  command  is  necessary  to  orient  TRIANGLE 
correctly. 

inserts  it  at  the  correct  olace.  The  student  again  asks  to 
FINISH  EDITING,  and  TURTLE  tells  him  that  he  has  completed  the 
task . 

4.1.2  WELL:  Decomposing  a  Complex  Figure 

'T'he  following  section  describes  TURTLE'S  tutoring  of  the 
more  complex  WELL  task,  and  focuses  on  how  TURTLE  recognizes 
standard  task  decompositions  and  helps  the  student  debug 
interfaces  and  figure  orientation. 
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The  session  begins  with  the  display  of  a  wishinq  well  (see 
Fiqure  2) ;  TURTLE  then  asks  the  student  to  enter  his  program, 
here  called  WELL. 


First  step?  TOP 
Next  step?  DRAW  200 
Next  step?  BOX 
Next  step?  QUIT 
OK,  WELL  is  defined! 


The  student  has  decomposed  the  WELL  into  three  steps:  his 

plan  is  to  draw  the  too  of  the  WELL  with  the  function  TOP,  the 

base  with  the  function  BOX,  and  the  POLE  with  the  open  coded  DRAW 

200.  TURTLE  now  determines  the  student's  Plan  from  this  program: 

Let  me  see  if  I  understand  your  program: 

The  function  TOP  draws  the  ROOF  of  the  WELL-TASK. 

The  code:  DRAW  200  draws  the  POLE  of  the  WELL-TASK. 

The  function  BOX  draws  the  BASE  of  the  WELL-TASK. 

Does  that  seem  correct?  YES 


By  identifving  the  student's  plan  for  the  program  —  in 
particular,  the  purposes  of  the  TOP  and  BOX  functions  and  the 
open  coded  DRAW  —  'f’URTLE  will  be  able  to  identify  errors  and 
offer  the  same  kinds  of  advice  that  it  did  in  the  TRIANGLE 
example.  TURTLE  now  asks  the  student  to  define  either  TOP  and 
BOX;  the  student  chooses  to  work  on  TOP  first. 


TOP  is  not  yet  defined.  Please  enter  the 
steps  in  TOP. 

First  step?  DRAW  100 
Next  step?  TURN  120 
Next  step?  DRAW  100 
Next  step?  TURN  120 
Next  step?  DRAW  100 
Next  step?  TURN  1R0 
Next  step?  QUIT 
OK,  TOP  is  defined! 


Note  that  this  is  identical  to  the  definition  of  TRIANGLE 
shown  previously,  except  for  the  addition  of  TURN  180  at  the  end 
of  the  proqram,  which  is  meant  to  put  the  the  turtle  in  the 
prooer  position  for  drawinq  the  WELL's  POLE.  (This  statement 
alone  is  not  sufficient;  this  error  will  be  corrected  later.) 
TURTLE  now  tries  to  determine  the  plan  underlvinq  the  student's 
top  proqram,  and  aqain  asks  which  of  the  three  possible  paths 
through  the  WELL's  triangle  corresponds  to  the  student's  olan: 


Did  you  to  intend  to  follow  the  path  B  C  A  B?  NO 

Did  you  to  intend  to  follow  the  path  A  B  C  A?  NO 

Did  you  to  intend  to  follow  the  path  C  A  B  C?  YES 


Using  this  plan,  TURTLE  determines  that  the  student's 
function  is  correct;  the  final  TURN  180  is  classified  as  part  of 
the  interface  between  the  TOP  and  the  (open  coded)  POLE.  TURTLE 
now  moves  the  student  on  to  the  definition  of  the  BOX  function: 


BOX  is  not  yet  defined.  Please  enter  the  steps  in  BOX. 

First  step?  DRAW  100 
Next  step?  TURN  90 
Next  step?  DRAW  100 
Next  step?  TURN  90 
Next  step?  DRAW  100 
Next  step?  TURN  90 
Next  step?  DRAW  100 
Next  step?  QUIT 
OK,  BOX  is  defined! 

As  before,  TURTLE  determines  the  path  the  student  planned  to  take 
through  the  box: 

Did  you  to  intend  to  follow  the  path  F  G  I  H  F?  YES 
and  notes  that  the  code  for  BOX  is  correct. 


Now  that  the  student  has  defined  the  three  components  of  the 
figure,  he  must  define  a  series  of  interface  steps  that  will 
prooerlv  position  these  oarts  of  the  program  with  respect  to  each 
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other.  TURTLE  first  checks  its  hypothesis  about  the  interface 
between  the  TOP  and  the  open  coded  POLE: 

Did  you  intend  to  interface  the  function  TOP  to  the 
step  DRAW  200  by  following  the  path  CD?  YES 

TURTLE  now  shows  the  student  whatever  interface  steps  are 
already  oresent  in  his  proqram  (here,  there  is  only  one  —  TURN 
180)  and  compares  them  to  the  correct  interface: 

Here  is  the  interface  you  defined  between  the  mainsteps 
in  the  function  TOP  and  the  step  DRAW  200: 

TURN  180 

Analysis  of  your  interface  between  the  function  TOP  and 
the  step  DRAW  200  in  WELL: 

Student  Correct  Commentary 

Code  Code 

TURN  180  TURN  180  Correct  interface  to  turn 

through  angle  BCD 

MOVE  50  Missing  interface  to  move 

over  line  segment  CD 

TURN  90  Missing  interface  to  turn 

through  angle  CDE 

TURTLE  now  guides  the  student  throuqh  these  modifications 
until  the  correct  interface  has  been  constructed.  Similar  steps 
are  taken  in  constructing  an  appropriate  interface  between  the 
open  coded  POLE  and  the  BOX.  TURTLE  then  checks  the  orientation 
of  the  entire  figure  and  detects  another  error:  although  the 
components  of  the  fiqure  itself  are  correct,  the  figure  as  a 
whole  is  at  an  incorrect  angle.  TURTLE  asks  the  student  about 
errors  in  this  orientation  — 


You  provided  no  setup  for  the  WELL-TASK. 

What  bugs,  if  any,  do  you  see  in  the  orientation  of  WELL? 
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Here,  the  student  requests  and  receives  a  hint  —  Another 
TURN  command  is  necessary  to  orient  WELL  correctly  —  and  enters 
the  necessary  step  —  TURN  30.  The  fiqure  is  now  constructed 
correctly,  and  the  session  ends. 

4.2  TURTLE:  Plan  Understanding  via  Analysis  by  Synthesis 

The  key  to  any  successful  tutorial  system  is  to  understand 

the  plan  that  the  student  has  generated  to  solve  the  problem  at 

hand.  In  this  project,  this  key  is  understanding  the  design  of 

the  student's  LOGO  program  and  the  function  of  each  of  the  steps 

of  the  proqram.  To  this  end,  TURTLE  exploits  the  constraints  of 

the  turtle  graphics  domain  and  the  restricted  subset  of  the  LOGO 

language  used  here:  TURTLE'S  first  step  in  identifying  the 

desiqn  of  a  student's  proqram  is  to  generate  the  entire  set  of 
10 

"viable"  paths  through  the  fiqure  and  to  compare  them  to  the 
oath  drawn  by  the  student's  program.  Since  plans  can  be 
associated  with  these  candidate  paths  when  they  are  generated, 
the  orohlem  of  identifying  the  student's  plan  is  reduced  to  the 
problem  of  findinq  the  path  that  offers  the  best  match  to  the 
student's  proqram. 

This  process  requires  three  classes  of  knowledqe  structures; 
the  remainder  of  this  section  describes  these  structures  and  how 
they  are  used  to  achieve  the  qoa.ls  of  appropriate  and  specific 
student  assistance,  turtle's  curriculum  structure  specifies  the 

— 

Viable  paths  are  those  with  a  limited  number  of  path  retraces 
(repeated  traversals  of  the  same  line  segment)  —  tvpicallv  no 
more  than  one  --  and  with  line  segments  that  are  broken  no  more 
than  once. 


i 
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textual  information  and  function  calls  that  comprise  the  tutorial 
session.  Its  task  networks  provide  thorough  descriptions  of  the 
figures  to  be  drawn  by  students  and  are  essential  to  TURTLE'S 
analysis  by  synthesis  evaluation  of  a  student's  program.  TURTLE 
uses  a  figure's  task  network  to  synthesize  a  number  of  programs 
capable  of  drawinq  this  figure.  Differences  between  these 
programs  and  the  student's  proposed  program  are  then  noted  by 
means  of  annotations  to  the  synthesized  programs.  These 
annotations  are  used  to  identify  the  student's  plan,  by  findinq 
the  synthesized  proqram  that  most  closely  matches  the  student's 
proqram,  and  to  guide  the  construction  of  hints,  by  means  of  a 
set  of  hint  generation  structures  that  detect  particular  error 
annotations  and  construct  hints  that  reflect  the  presence  of 
these  errors. 

4.2.1  Curriculum  Structure 

For  each  problem  in  the  curriculum  that  is  to  be  presented 
by  TURTLE,  two  functions  are  defined:  a  presentation  function 
that  will  carry  out  the  actual  tutoring  of  the  problem,  and  a 
successor  function  that  will  determine  the  next  problem  that 
should  be  presented  to  the  student.  An  executive  function  then 
retrieves  and  executes  the  presentation  function  for  the  selected 
problem  (thereby  carrvinq  out  this  problem's  tutorial  session) , 
and,  by  executing  the  successor  function,  selects  the  next 
problem  that  will  be  given  to  the  student.  The  successor 
function  can,  in  principle,  base  this  decision  on  such  factors  as 
the  student's  performance  on  the  current  and  previous  problems. 
However,  at  this  early  stage  in  'turtle's  development,  the 
problems'  successor  functions  simply  pass  the  student  through  the 
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TRIANGLE,  TREE,  WELL,  and  NAPOLEON  tasks,  in  that  order. 

4.2.2  Task  Representation 

As  described  previously,  TURTLE  qenerates  all  viable  paths 
through  a  fiqure  and  then  matches  the  student's  program  against 
these  paths.  To  generate  these  paths,  TURTLE  must  have  a 
representation  of  the  figure  that  is  explicit  enough  to  support 
path  generation,  yet  general  enough  to  allow  for  errors  in  the 
student's  proqram  and  for  variations  in  the  size  and  orientation 
of  the  figure  drawn  by  the  student. 

TURTLE  uses  a  svstem  of  task  networks  to  organize  this 
information.  The  task  networks  for  the  TRIANGLE  and  WELL  figures 
are  shown  in  Tables  7  and  8. 


These  networks  are  frame-like  units  with  the  following 
slots : 

-TASK-NAME:  the  name  of  the  task. 

-FIGURE-GEOMETRY:  a  specification  of  the  vectors  and 
angles  that  comprise  the  figure.  This  slot  specifies: 

-VERTICES:  the  points  of  the  figure  that  are  to  be 
connected . 

-CONNECTIONS:  the  line  segments  that  connect  the 
figure's  vertices.  This  slot  also  contains 
information  about  the  size  and  possible 
subdivision  of  the  line  segments. 

-INTERFACES:  the  size  of  angles  formed  by  connected 
line  segments. 

-STARTING-POINTS:  the  vertices  that  may  be  used  to 
begin  the  figure's  construction. 

-DESIRED-ORIENTATION:  the  orientation  of  a  particular 
line  segment,  by  which  the  orientation  of  the  entire 
figure  can  be  determined. 


-POSSIBLE-NAMES:  a  list  of  possible  names  for  the  figure 
and  anv  subfigures. 
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Table  7: 

turtle's  task  network  for  TRIANGLE. 
References  to  vertices  correspond  to  those  of  the 
figures  shown  in  Figure  2. 


TASK-NAME:  TRIANGLE 

FIGURE-GEOMETRY : 

VERTICES : 

(a  b  c) 

CONFECTIONS : 

((join  a  b) 

via  (line-segment  1  x) ) 
( (join  a  c) 

via  (line-segment  1  x) ) 
( (join  b  c) 

via  (line-segment  1  x) ) 
(default-for  x  100) 

INTERFACES : 

(cab  angle  60) 

(h  c  a  angle  60) 

(a  b  c  angle  60) 

STARTING-POINTS: 

(start-at  (a  b  c) ) 

DES I RED-ORIENTATION : 

(orientation  a  b  should-be  150) 

EXPECTED-DECOMPOSITIONS : 

(triangle  ->  open-code) ) 


EXAMPLE-SOLUTION : 

((turn  30) 

(draw  1  x) 

(turn  120) 

(draw  1  x) 

(turn  120) 

(draw  1  x) ) 

( (setup-for-task) 
(draw-line-segment  c  a) 
(turn-through-anqle  cab) 
(draw-line-segment  a  b) 
(turn-through-anqle  a  b  c) 
(draw-line-segment  b  c) ) 
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Table  8 : 

TURTLE^s  task  network  for  WELL. 


TASK-NAME:  WELL 
FIGURE-GEOMETRY : 

VERTICES: 

(abcdefqhi) 

CONNECTIONS : 

((join  a  b)  ;Roof  of  the  well 

via  (line-seqment  2  x) ) 

( ( join  a  c) 

via  (line-segment  2  x) ) 

( (join  b  c) 

via  (line-seqment  2  x) 
contains  ((b  d)  (c  d) ) 
subdivided-by  d) 

( ( join  b  d) 

via  (line-seqment  1  x) 
contained-in  b  c) 

( (join  d  c) 

via  (line-segment  1  x) 
contained-in  b  c) 

;Pole  of  the  well 

( (join  d  e) 

via  (line-seqment  1  z) ) 

;Base  of  the  well 

(  (join  h  f) 

via  (line-segment  2  v) 
subdivided-by  e 
contains  ( (h  e)  (e  f ) ) ) 

( (join  f  q) 

via  (line-seqment  2  v) ) 

(  (join  q  i) 

via  (line-seqment  2  v) ) 

( (join  i  h) 

via  (line-seqment  2  v) ) 

( (join  h  e) 

via  (line-seqment  1  v) 
contained-in  h  f) 

((join  e  f) 

via  (line-seqment  1  v) 
contained-in  h  f) 

(default-for  (x  100)  (v  100)  (z  200)) 
INTERFACES : 

(d  c  a  anqle  60)  (a  b  d  angle  60) 

(cab  anqle  60)  (b  c  a  anqle  60) 

(a  b  c  anqle  60)  (e  f  q  angle  90) 

(i  h  e  anqle  90)  (q  i  h  anqle  90) 

(f  q  i  anqle  90)  (h  f  q  angle  90) 
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(i  h  f  anqle  90)  (d  e  h  angle  90) 

(fed  anqle  90)  (e  d  b  anqle  90) 

(c  d  e  anqle  90)  (c  d  b  anqle  180) 

(bed  angle  0)  (c  b  d  anqle  0) 

(h  e  f  anqle  180)  (f  h  e  anqle  0) 

(h  f  e  angle  0) 

STARTING-POINTS : 

(abedefghi)) 

DESIRED-ORIENTATION: 

(orientation  i  q  should-be  90) 

POSSIBLE-NAMES: 

(roof  is-also-called  (triangle  roof  ro  tri  tr  top)) 
(pole  is-also-called  (pole  po  line  li  line-segment 

vect  middle  mid) ) 

(base  is-also-called  (base  ba  square  sq  squ  bottom 

bot  bo) ) 

(tree  is-also-called  (tree)) 

EXPECTED-DECOMPOSITIONS : 

(well  ->  roof  oole  base 
(Vertices : 

( (roof  abed) 

(pole  d  e) 

(base  h  e  f  q  i))) 
(Termination-Points : 

(((roof  ending-point)  d) 

( (pole  starting-point)  d) 

( (pole  endinq-point)  e) 

((base  starting-point)  e) ) ) ) 

(well  ->  base  pole  roof 
(Vertices : 

( (roof  abed) 

(pole  d  e) 

(base  h  e  f  g  i) ) ) 
(Termination-Points : 

(((roof  starting-point)  d) 

((pole  endinq-point)  d) 

( (pole  starting-point)  e) 

((base  endinq-point)  e) ) ) ) 

(well  ->  tree  base 
(Vertices : 

( ( tree  a  b  c  d  e) 

(base  h  e  f  q  i))) 
(Termination-Points : 

(((tree  endinq-ooint)  e) 

((base  startinq-Doint)  e)))) 

(well  ->  base  tree 
(Vertices : 

( (tree  a  b  c  d  e) 

(base  h  e  f  g  i) ) ) 
(Termination-Points : 

(((tree  starting-point)  e) 

((base  endinq-point)  e) ) ) ) 
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(well  ->  ooen -code) 

EXAMPLE-SOLUTION : 

((turn  270) 

(draw  1  x) 

(turn  120) 

(draw  2  x) 

(turn  120) 

(draw  2  x) 

(turn  120) 

(draw  1  x) 

(turn  270) 

(draw  1  z) 

(turn  90) 

(draw  1  v) 

(turn  270) 

(draw  2  v) 

(turn  270) 

(draw  2  v) 

(turn  270) 

(draw  2  v) 

(turn  270) 

(draw  1  y) ) 

( (setup-for-task) 

(draw-line-segment  d  c) 

( turn-through-angle  d  c  a) 

(draw-line-segment  c  a) 

(turn-through-angle  cab) 

(draw-line-segment  a  b) 

(turn-through-angle  a  b  d) 

(draw-line-segment  b  d) 

(turn-through-angle  b  d  e) 

(draw-line-segment  d  e) 

(turn-through-angle  d  e  h) 

(draw-line-segment  e  h) 

(turn-through-angle  e  h  i) 

(draw-line-segment  h  i) 

(turn-through-angle  h  i  g) 

(draw-line-segment  i  g) 

(turn-through-angle  i  g  f) 

(draw-line-segment  g  f) 

(turn-through-angle  g  f  i) 

(draw-line-segment  f  e) ) ) ) 

(default-bindings  well  ((X  100.0)  (Y  100.0)  (7  200.0))) 
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-EXPECTED-DECOMPOSITIONS:  descriptions  of  how  the  figure 
miqht  be  decomposed.  For  each  of  the  figures  that  can 
be  decomposed  (TRIANGLE  cannot) ,  this  slot  lists  the 
sets  of  vertices  that  comprise  these  subfigures  and  the 
acceDtable  beqinninq  and  endinq  points  for  the 
subfigures . 

-EXAMPLE-SOLUTION:  a  LOGO  program  that,  once  exact 
values  for  the  line  seqments  have  been  determined,  will 
draw  the  fiqure  correctly.  This  structure  also 
contains  qeneral  descriotions  of  the  function  of  each 
of  the  statements  of  the  proqram,  such  as 
DRAW-LINE-SEGMENT  and  TURN-THROUGH-ANGLE . 

4.2.3  Program  Synthesis 

These  task  networks  Drovide  the  descriptive  information 
TURTLE  uses  to  qenerate  the  viable  oaths  through  a  fiqure  and  the 
LOGO  proqrams  that  correspond  to  these  paths.  The  result inq 
paths  can  then  be  matched  against  the  student's  code,  errors  can 
be  detected  and  diagnosed,  and  appropriate  hints  can  be  qiven. 

This  path  generation  depends  upon  the  specification  of  the 
figure's  STARTiNG-POINTs  and  CONNECTIONS.  TURTLE  generates  all 
viable  paths  from  each  of  the  fiqure's  starting  points,  as 
determined  by  the  connections  between  the  fiqure's  vertices.  In 
TRIANGLE'S  task-network,  since  all  three  of  the  triangle's 
vertices  are  specified  as  startinq  points,  oaths  through  the 
trianqle  would  be  generated  from  vertices  A,  B,  and  C.  The 
connection  information  would  then  guide  the  oath  construction 
process:  two  paths  would  be  built  from  vertex  A,  based  on  the 
line  segments  between  vertices  A  and  B  and  between  A  and  C.  These 
oaths  are  then  expanded  in  a  similar  way:  for  instance,  the  "A  -> 
R"  path  is  expanded  to  "A  ->  B  ->  C"  via  the  line  segment  from  R 
to  C.  This  generation  process  operates  under  three  constraints. 
First,  all  the  fiqure's  line  segments  must  appear  in  the  final 
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path.  Second,  a  line  segment  can  be  traversed  more  than  once, 
but  only  one  of  the  traversals  can  be  a  DRAW;  the  rest  must  be 
retraces  that  MOVE  over  the  already  drawn  line.  Third,  no  more 
than  a  fixed  number  of  retraces  may  take  place  —  if  retraces 
were  not  limited,  an  infinite  number  of  open  coded  solutions 
would  exist  for  even  the  simplest  figures.  Our  experiments  with 
TURTLE  limited  the  number  of  acceptable  retraces  to  one. 

This  technique  produces  six  possible  paths  for  the  TRIANGLE 
f iqure : 

A  ->  B  ->  C  ->  A  A  ->  C  ->  B  ->  A 

R  ->  C  ->  A  ->  B  R  ->  A  ->  C  ->  B 

C  ->  A  ->  B  ->  C  C  ->  B  ->  A  ->  C 

These  paths  are  then  converted  into  LOGO  code  by  referring  to  the 

task  network's  specifications  of  line  seqments  and  interface 
anqles.  Consider  the  path  "A  ->  B  ->  C  ->  A".  The  first  step  in 
this  path,  "A  ->  B" ,  corresponds  to  the  CONNECTIONS  entry: 

((JOIN  A  B)  VIA  (LINE-SEGMENT  1  X)) 

(see  Tahle  7)  and  a  DRAW  is  qenerated.  Since  the  absolute  size 
of  the  figure  that  will  be  drawn  by  the  student  cannot  be 
predicted,  the  lenqth  of  this  line  seqment  is  specified  in  terms 
of  a  constant  and  a  variable.  The  "1  X"  in  the  "(LINE-SEGMENT  1 
X) "  structure  indicates  the  desired  lenqth  of  the  line  segment  as 
the  product  of  the  scale  factor,  1,  and  a  variable,  X;  the 
oseudo-LOGO  statement  "DRAW  1  X"  is  qenerated.  These 
constant-variable  pairs  allow  turtle  to  interpret  student 
oroqrams  of  anv  absolute  size;  the  only  requirement  for  a  correct 
program  is  that  the  relative  sizes  of  the  figure  correspond  to 


those  defined  in  the  task  network.  Default  values  for  these 
variables  are  specified  so  that  TURTLE  can  generate  executable 
LOGO  code  from  these  structures  before  explicit  line  segment 
lengths  have  been  specified. 


The  next  step  in  this  Dath  is  "B  ->  C" ;  here,  TURTLE  should 
build  the  code  not  only  for  the  DRAW  from  B  to  C,  but  also  the 
TURN  that  will  draw  this  line  segment  in  the  proper  direction. 
This  TURN  through  the  anqle  ABC  is  derived  from  the  angle's 
INTERFACE  specification: 

(ABC  ANGLE  60) 

The  instruction  "TURN  120"  results:  the  60  degree  interior  angle 
of  the  triangle  is  generated  by  turning  the  turtle  through  the 
120  degree  exterior  angle.  The  remaining  instructions  for  this 
Dath  and  for  the  other  five  paths  are  generated  in  this  way, 
producing  six  solution  structures: 


( SOLUTION 1 

(A  B  C  A) 
(DRAW  1  X) 
(TURN  120) 
(DRAW  1  X) 
(TURN  120) 
(DRAW  1  X)) 

(SOLUTION4 

(A  C  B  A) 
(DRAW  1  X) 
(TURN  240) 
(DRAW  1  X) 
(TURN  240) 
(DRAW  1  X)) 


(SOLUTION2 

(B  C  A  B) 
(DRAW  1  X) 
(TURN  120) 
(DRAW  1  X) 
(TURN  120) 
(DRAW  1  X)) 

(SOLUTIONS 

(C  B  A  C) 
(DRAW  1  X) 
(TURN  240) 
(DRAW  1  X) 
(TURN  240) 
(DRAW  1  X)) 


(SOLUTION 3 

(C  B  A  C) 
(DRAW  1  X) 
(TURN  120) 
(DRAW  1  X) 
(TURN  120 ) 
(DRAW  1  X)) 

(SOLUTION6 

(B  A  C  B) 
(DRAW  1  X) 
(TURN  240) 
(DRAW  1  X) 
(TURN  240) 
(DRAW  1  X) ) 


4.2.4  Program  Recognition 

Once  these  solution  structures  have  been  generated,  a 


student's  nrooosed  program  can  be  matched  against  these 


structures.  Since  an  exact  match  will  occur  only  if  the  proqram 
is  correct,  partial  matchinq  techniques  are  used  to  permit  and 
identify  differences  between  the  proposed  and  correct  solutions. 

Recall  the  illustrated  TRIANGLE  task  once  again:  the 
student's  proposed  proqram  was: 

DRAW  200 
TURN  60 
DRAW  200 

This  proqram  must  be  matched  against  the  six  proposed  solutions 
shown  above.  The  first  step  in  this  process  is  to  build  a 
matchinq  structure  from  this  proqram;  this  is  done  by  reolacinq 
each  arqument  of  the  program's  TURN  statements  with  position 
variables,  which  can  match  anv  anqle  value  —  includinq  the  valm 
of  incorrect  turns  —  and  by  insertinq  before  and  after  each 
proqram  statement  segment  variables,  which  can  match  anv  number 
of  statements  and  so  can  identifv  statements  the  student  has 
incorrectlv  entered  or  omitted  from  his  proqram.  This  procedure 
produces  the  structure: 

SS0  (DRAW  200)  SSI  (TURN  ? ANGLE)  $S2  (DRAW  200)  $S3) 
which  must  be  matched  aqainst  the  proposed  solutions,  perhaps 
SOLUTION 1: 

(DRAW  1  X)  (TURN  120)  (DRAW  1  X)  (TURN  120)  (DRAW  1  X) 

'’’his  match  will  succeed,  as  the  result  of  four  separate 
steps.  First,  the  program's  first  DRAW  matches  the  solution's 
first  DRAW,  qivinq  X  —  the  variable  used  to  soecifv  the  lengths 
of  line  segments  —  a  value  of  200.  Note  that  the  matcher 


attempts  to  match  DRAWS  wherever  possible,  rather  than  letting 
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DRAWs  be  matched  by  segment  variables.  As  a  result,  the  segment 
variable  SSO  receives  no  value.  Second,  the  program's  TURN 
matches  the  solution's  TURN,  with  the  position  variable  7VALUE 
set  to  120  (not  60,  as  in  the  student's  original  program).  Since 
this  successfully  matched  TURN  immediately  followed  the 
previously  matched  DRAW,  $S1  receives  no  value.  Third,  the 
program's  second  DRAW  matches  the  solution's  second  DRAW;  the 
success  of  this  match  results  in  $S2  receiving  no  value.  Fourth, 
since  the  student's  proqram  ends  with  this  second  DRAW,  the 
segment  variable  $S3  is  matched  to  the  remaining  statements  in 
the  solution  —  (TURN  120)  and  (DRAW  1  X). 

This  technique  is  used  to  match  the  student's  program 
against  the  proposed  solutions.  Each  of  these  matches  is  then 
given  penalty  points  that  determine  the  match's  "badness  of  fit"; 
the  following  ad  hoc  rules  are  used  to  award  these  points: 

-Missing  steps:  1  point. 

-Incorrect  TURNs: 

-If  the  sum  of  the  angles  is  180  degrees,  .25 
point.  This  rule  will  trap  the  common  error  of 
specifying  turns  by  their  interior,  and  not 
exterior,  angles. 

-If  the  proposed  and  correct  turns  are  both 
clockwise  (less  than  180  degrees)  or  both 
counterclockwise  (between  180  and  360  degrees),  .5 
point . 

-Otherwise,  1  point. 

This  method  awards  R0LUTI0N1  above  a  score  of  2.5  points:  2 
points  for  the  two  missing  steps,  and  .5  point  for  the  TURN 
statement:  the  exact  angle  of  the  turn  is  wrong,  but,  like  the 
corresponding  statement  in  the  solution,  it  is  in  a  clockwise 
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direction.  Since  S0LUTI0N1,  SOLUTION?,  and  SOLUTIONS  all  use 
clockwise  turns,  they  all  will  receive  scores  of  2.5.  Similarly, 
since  S0LUTI0N4,  SOLUmION  5,  and  S0LUTI0N6  are  identical,  but  use 
counterclockwise  turns  to  solve  the  problem,  they  will  receive 
scores  of  3  points.  The  incorrect  TURN  in  these  solutions 
receives  a  penalty  score  of  1  point. 

SOLUTION 1 ,  SOLUTION?,  and  S0LUTI0N3  are  retained  as 
candidates  for  the  correct  interpretation  of  the  student's  plan, 
since  they  share  the  score  that  indicated  the  best 
oroqram-solution  match.  However,  turtle  cannot  determine  which 
of  these  three  solutions  is  correct  without  knowing  which  of  the 
vertices  of  the  triangle  was  used  by  the  student  as  the  starting 
point  of  his  program.  This  problem  can  be  resolved  only  by 
asking  the  student  which  of  the  three  hypothesized  paths 
corresponds  to  his  own: 


Did  you  to  intend  to  follow  path  C  A  B  C?  NO 

Did  you  to  intend  to  follow  path  B  C  A  B?  NO 

Did  you  to  intend  to  follow  path  A  B  C  A?  YES 

In  this  way,  the  correct  solution  —  here,  S0LUTI0N1  —  can 

be  identified,  and  the  plan  corresponding  to  this  solution  can  be 
used  to  analyze  and  correct  the  errors  in  this  program. 

4.2.5  Program  Decomposition 

The  simplicity  of  the  TRIANGLE  task  makes  the  identification 
of  the  student's  plan  very  straightforward.  However,  the 
technique  described  above  will  be  less  successful  as  the  tasks  to 
which  it  is  applied  become  more  complex  and  the  number  of 
possible  paths  through  these  figures  increases.  This  problem  can 
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he  avoided  by  encouraging  the  student  to  decompose  a  large 
problem  into  a  set  of  smaller  problems.  Problem  decomposition  is 
typically  good  programming  style  and  would  simplify  TURTLE  s  task 
considerably.  It  would  have  to  do  the  kinds  of  analysis  shown 
above  only  on  simple  figures  such  as  triangles  and  squares,  where 
the  number  of  viable  paths  is  small.  However,  this 
simplification  comes  at  a  price:  'TORfLE  must  be  able  to  identify 
how  the  student  is  decomposing  the  problem,  so  that  proper 
interpretations  can  be  qiven  to  the  proqram  segments  that  carry 
out  the  individual  tasks. 

TURTLE  interprets  a  student's  decomposition  in  much  the  same 
way  it  determines  the  plan  underlying  an  open  coded  oroqram.  Its 
strategy  is  based  on  two  assumptions  about  how  people  will  solve 
these  problems: 

-Peoole  will  decompose  a  problem  that  draws  a  complex 
fiqure  into  a  set  of  functions  that  draw  regular 
fiqures  such  as  triangles,  rectangles,  and  circles. 

-Peoole  will  use  mnemonic  names  for  the  functions  that 
carry  out  particular  subparts  of  the  problem:  a 
function  that  draws  the  square  base  of  the  well  will 
orobablv  be  called  BASE  or  SQUARE. 

TURTLE  exploits  these  assumptions  in  the  following  ways. 

The  first  assumption  implies  that  TURTLE  need  anticipate  only 
those  decompositions  made  up  of  regular  figures.  Descriptions  of 
these  orobable  decompositions  can  then  be  included  in  a  fiqure  s 
task  network;  the  expected  decompositions  for  WELL  include 
structures  such  as  the  followinq  (the  full  set  of  expect  d 
decompositions  can  be  found  in  WELL's  task  network  in  Fiqure  8): 


d 
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EXPECTED-DECOMPOSITIONS : 

(WELL  ->  ROOF  POLE  BASE 
(VERTICES : 

( ( ROOF  A  B  C  D) 

(POLE  D  E) 

(BASE  H  E  F  G  I) ) ) 
(TERMINATION-POINTS : 

(((ROOF  ENDTNG-POINT)  D) 

( (POLE  STARTING-POINT)  D) 

( (POLE  ENDING-POINT)  E) 

( (BASE  STARTING-POINT)  E) ) ) ) 

This  structure  states  that  the  WELL  consists  of  a  ROOF,  a 
POLE,  and  a  BASE.  The  ROOF  is  drawn  by  vertices  A,  B,  C,  and  D 
(see  Fiqure  2) ,  the  POLE  by  vertices  D  and  E,  and  the  BASE  by 
vertices  E,  F,  G,  H,  and  I.  The  WELL  can  also  be  drawn  as  a  'T’REE 
and  a  BASE;  a  structure  that  represents  this  decomposition  is 
also  included  in  WELL's  task  network.  TURTLE  will  use  some 
subset  of  these  four  names  to  internally  describe  how  the  student 
has  decomposed  the  WELL  problem.  All  programs  involving 
decomposition  will  be  determined  to  be  some  collection  of  the 
subf iqures  ROOF,  POLE,  BASE,  and  TREE. 

The  identification  of  students"  decompositions  would  be 
simple  if  students  used  only  these  four  names  for  the  subfigure 
functions.  This,  of  course,  is  not  the  case.  However,  since 
people  will  probably  give  the  subfigure  functions  mnemonic  names, 
the  number  of  probable  function  names  is  limited.  Like  the 
fiqure  decompos it  ions ,  these  names  can  be  enumerated  and  included 
in  the  figure's  task  network;  the  names  recognized  by  TURTLE  for 
the  functions  used  to  draw  the  possible  subparts  of  WELL  —  ROOF, 
POLE,  tree,  and  BASE  —  are  structured  as  follows: 
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POSSIBLE -NAMES: 

(ROOF  I S-ALSO-C ALLED 

(POLE  I S-ALSO-C ALLED 

(BASE  IS- ALSO-CALLED 
(TREE  IS- ALSO-CALLED 


(ROOF  RO  TRIANGLE  TRI  TR 
TOP)  ) 

(POLE  PO  LINE  LI 

LINE-SEGMENT  VECT 
MIDDLE  MID) ) 

(BASE  BA  SQUARE  SQ  SQU 
BOTTOM  BOT  BO) ) 

(TREE) ) 


These  structures  are  then  used  to  identify  the  decomposition 
used  by  a  student.  Recall  the  proposed  proqram  for  the  WELL  task 
(Section  4.1.2) : 


TOP 

DRAW  200 
BOX 


The  decomposition  represented  by  this  proqram  is  determined 
by  converting  the  proqram  into  a  matching  structure  and  comparing 
it  to  the  possible  decompositions.  The  construction  of  this 
matching  process  requires  three  steps: 

-Each  student  function  name  that  appears  in  the 
POSSIBLE-NAMES  structure  is  replaced  by  the  concept 
under  which  the  function  name  is  indexed.  By  this 
rule,  the  student  function  name  TOP  is  converted  to 
ROOF. 

-Each  function  name  that  does  not  appear  in  the 
POSSIBLE-NAMES  structure  is  converted  to  a  position 
variable:  since  BOX  is  not  a  name  known  to  TURTLE,  it 
is  replaced  by  the  position  variable  ?BOX. 

-All  strings  of  LOGO  function  calls  are  replaced  bv 
segment  variables:  the  DRAW  200  statement  is  replaced 
by  $S0 . 


This  conversion  process  produces  the  matching  structure 

(ROOF  $S0  ?BOX) 

which  can  be  matched  against  the  set  of  possible  decompositions 
of  the  WELL;  as  before,  SS0  can  match  anv  string  of  statements, 
and  ?BOX  can  match  any  single  item.  The  best  match  comes  from 


*  A 


the  decomposition 


(ROOF  POLE  BASE) 

which  leads  to  TURTLE'S  interpretation  of  the  student's  program; 

Let  me  see  if  I  understand  how  you  are  going  to  solve 

the  WELL-TASK : 

The  function  TOP  draws  the  ROOF  of  the  WELL-TASK. 

The  code:  DRAW  200  draws  the  POLE  of  the  WELL-TASK. 

The  function  BOX  draws  the  BASE  of  WELL-TASK. 

Does  that  seem  correct?  YES 

Note  that  the  matching  process  will  fail  if  the  student 
writes  a  program  with  function  names  that  are  recognized  by 
TURTLE,  but  that  appear  in  an  unexpected  order,  such  as  ROOF  -> 
BASE  ->  POLE.  When  given  such  a  program,  TURTLE  will  ask  the 
student  to  define  the  paths  to  be  drawn  by  each  of  these 
programs;  if  these  completely  describe  the  figure  to  be  drawn,  it 
will  proceed  with  this  decomposition.  TURTLE  also  allows  the 
student  to  define  new  decompositions  through  a  similar  technique 
of  askinq  the  student  to  define  the  paths  taken  by  the  new 
functions . 

4.2.6  Specifying  Setup  Orientations  and  Interfiqure  Interfaces 

The  discussion  thus  far  has  concentrated  on  the  generation 
of  programs  that  correctly  draw  the  basic  figures  that  make  up 
the  tasks  discussed  here  —  trianqles,  squares,  and  the  like. 
There  are  two  further  tasks,  however,  that  affect  the  correctness 
of  a  LOGO  proqram.  These  are  concerned  with  the  spatial 
orientation  of  the  program's  components. 

As  nou<jd  in  the  sample  TRIANGLE  session,  the  following  code 
generates  a  correct  triangle: 


DRAW  100 
TURN  120 
DRAW  100 
TURN  120 
DRAW  100 

Since  the  student  began  this  figure  at  vertex  A,  the  above 
code  will  produce  a  triangle  that  "points"  to  the  right.  Since 
the  turtle  begins  a  session  facing  due  north,  the  first  DRAW  also 
goes  north.  Facing  north  is  only  one  of  many  oossible 
orientations,  however,  and  is  in  fact  incorrect.  As  shown  in 
Figure  2,  the  generated  triangle  must  point  upward.  The  program 
needs  an  additional  "setup"  step  at  the  beginning  of  the  program 
—  TURN  30  —  to  place  the  triangle  in  its  proper  orientation. 

The  figure's  orientation  is  specified  in  TRIANGLE'S  task 
network  as  its  DESIRED-ORIENTATION: 

(ORIENTATION  A  B  SHOULD-BE  150) 

This  form  states  that  the  line  segment  from  A  to  B  should  form  a 
150  degree  angle  with  a  horizontal  line.  Since  the  angles  drawn 
in  turtle-graphics  systems  are  assumed  to  be  rigid,  the  entire 
orientation  of  an  otherwise  correct  figure  can  be  determined  by 
examining  this  one  segment. 

TURTLE  determines  the  current  orientation  of  this  line 
segment  by  generating  (via  the  figure's  task  network  CONNECTIONS) 
a  nath  from  the  line  segment  backward  to  the  starting  point  of 
the  figure,  which  the  student  identified  when  he  verified 
TURTLE'S  interpretation  of  his  plan.  Since  the  line  segment 
drawn  at  the  very  beginning  of  the  figure  will  have  an 
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orientation  of  zero  degrees ,  and  the  angles  between  the  line 
seqments  that  make  up  this  path  are  also  specified  in  the  task 
network,  the  actual  orientation  of  AB  can  be  computed.  TURTLE 
can  then  determine  the  size  of  the  "setup"  turn  from  the 
difference  between  the  desired  and  actual  orientations  of  AB; 
here,  that  difference  is  30  degrees.  The  student  can  then  be 
tutored  on  the  construction  of  this  interface. 

The  second  orientation  problem  is  caused  by  the 
decomDosit ion  of  a  proqram  into  simpler  tasks.  As  shown  earlier, 
the  WELL  problem  can  be  loqicallv  subdivided  into  three  subtasks 
—  a  square,  a  line7  and  a  triangle.  Both  of  the  functions  and 
the  open  coded  line  segment  written  by  the  student  in  the  sample 
session  (Section  4.1.2)  were  individually  correct;  however,  these 
functions,  when  called  one  after  another,  produce  the  incorrect 
drawinq  shown  in  Figure  3. 

One  error  in  this  figure  is  that  no  setup  orientation  is 
present;  a  more  significant  problem,  however,  is  that  the  figures 
are  not  interfaced  properly;  the  POLE  leaves  the  TOP  and  enters 
the  BOX  at  the  wronq  places.  Although  the  student  whose  session 
appears  in  Section  4.1.2  tried  to  define  an  interface  between  TOP 
and  the  DRAW  —  via  the  TURN  180  at  the  end  of  TOP  —  additional 
statements  are  needed  between  all  three  of  these  statements  to 
insure  that  the  three  figures  are  properly  positioned. 
Unfortunately,  no  one  set  of  interface  statements  exists  that 

_ 

unless  the  proqram  beqan  with  a  TURN  statement,  in  which  case 
the  first  line  segment  will  have  an  orientation  corresponding  to 
the  angle  of  the  'T’URN. 


might  be  identified  and  included  in  the  figure's  task  network  as 
were  figure  decompositions,  because  the  student's  design  of  the 
subfigures  will  determine  the  steps  required  to  interface  these 
figures.  TURTLE  must  therefore  wait  until  the  student  has 
defined  the  startinq  and  ending  points  of  his  subfigures,  and 
then  generate  interface  steps  that  are  appropriate  for  these 
functions . 

When  the  interface  between  two  subfiqures  is  being 
generated,  TURTLE  has  determined  how  the  student  has  decomposed 
the  task,  and  so  knows  which  vertices  are  drawn  by  which 
subfigures.  In  the  example  above,  TURTLE  has  determined  that  TOP 
follows  the  path  C  ->  A  ->  B  ->  C,  the  DRAW  that  creates  the 
WELL's  POLE  moves  from  D  to  E,  and  BOX  follows  the  path  F  ->  G  -> 
I  ->  H.  'The  interface  steps  between  TOP  and  the  DRAW  must 
therefore  move  the  turtle  from  C  to  D  (from  the  ending  point  of 
TOP  to  the  startinq  point  of  the  DRAW)  and  position  it  properly 
for  the  subsequent  DRAW.  Similarly,  the  interface  steps  between 
the  DRAW  and  BOX  must  move  from  E  to  F  and  position  the  turtle 
properly  for  the  drawing  of  the  BOX.  These  paths  can  be 
generated  in  much  the  same  way  as  the  paths  that  draw  the 
individual  figures:  by  generating  all  sequences  steps  through  the 
fiqure,  subject  to  the  constraints  on  retraces  described  in 
Section  4.2.3.  TURTLE  generates  the  paths  C  ->  D  and  C  ->  A  ->  B 
->  D  as  possible  interfaces  between  vertices  C  and  D.  These  paths 
are  then  converted  to  LOGO  code,  again  in  much  the  same  way  as 
the  original  proqrams  —  by  usinq  the  specifications  of  line 
segments  and  anqles  in  the  task's  f igure-qeometry  to  convert  the 
paths  into  an  appropriate  set  of  TURNS  and  DRAWS.  Note  that 


TURTLE  must  refer  to  the  steps  immediately  before  and  after  the 
interface  steps  to  determine  the  initial  and  final  TURNS.  To 
correctly  turn  onto  the  path  from  C  to  D,  TURTLE  must  know  the 
orientation  of  the  turtle  upon  reaching  vertex  C,  which  would 
depend  upon  whether  the  student  drew  TOP  along  the  path  C  ->  A  -> 
B  ->  C  or  the  path  C  ->  B  ->  A  ->  C.  Since,  in  this  example, 
TURTLE  knows  that  the  student  followed  the  path  C  ->  A  ->  B  ->  C 
in  TOP,  the  following  possible  interface  solutions  result: 


(S0LUTI0N1 

(C  D) 

(TURN  180) 
(MOVE  1  X) 
(TURN  90)) 


;  turn  through  angle  DCD 
;  move  over  line  segment  CD 
;  turn  through  angle  CDE 


(SOLUTION2 

(C  A  B  D) 
(TURN  120) 
(MOVE  2  X) 
(TURN  120) 
(MOVE  2  X) 
(TURN  120) 
(MOVE  1  X) 
(TURN  270)) 


turn  through  angle  ACD 
move  over  line  segment  AC 
turn  through  angle  CAB 
move  over  line  segment  ab 
turn  through  angle  ARD 
move  over  line  segment  BD 
turn  through  angle  BDE 


The  student's  interface  is  matched  against  these  two 
possible  solutions.  As  before,  points  are  awarded  that  evaluate 
the  quality  of  the  match  (although  a  different  system  for 
awarding  point  values  is  used  here) ,  and  the  solution  with  the 
better  (i.e.,  lower)  score  is  selected.  Here,  SOLUTIONl  is  the 
better  solution  —  its  TURN  180  matches  the  student's  interface 
exactly,  and  its  number  of  steps  is  much  closer  to  that  of  the 
student's  plan  —  and  so  will  be  used  to  debug  the  student's 


interface 


4.2.7  Program  Annotation  and  Hint  Generation 

Once  the  plans  underlying  the  student's  programs  and 
interfaces  have  been  identified,  TURTLE  tries  to  identify  any 
errors  that  may  be  oresent  in  the  program.  This  identification 
is  a  two-step  process.  First,  patterns  are  constructed  that 
describe  the  intent  of  each  statement  of  the  proqram  and  whether 
that  statement  achieves  that  intent;  second,  these  patterns  are 
matched  against  predefined  structures  that  reveal  the  presence  of 
certain  kinds  of  errors. 

Program  errors  are  interpreted  by  matching  the  student's 
actual  program  to  the  program  that,  as  hypothesized  by  TURTLE, 
corresponds  to  the  student's  plan.  The  result  of  this  match  is  a 
set  of  PROGRAM-STEP  structures,  one  for  each  statement  of  the 
correct  program.  These  structures  are  of  the  form: 

(PROGRAM-STEP  number 

step-representation 

evaluation 

intended-action) 

A  program-step's  NUMBER  is  simply  the  ordinal  number  of  the 
steo  in  the  correct  orogram.  The  STEP-REPRESENTATION  consists  of 
an  arbitrary  and  unique  label  for  the  steD,  followed  by  the 
correct  statement  itself.  The  EVALUATION  states  whether  the  step 
is  CORRECT  or  INCORRECT  in  the  student's  program,  or  whether  it 
is  MISSING.  The  INTENDED-ACTION  is  a  general  description  of  the 
statement's  function  —  such  as  (DRAW-LINE-SEGMENT  B  C)  —  which 
is  obtained  from  the  EXAMPLE-SOLUTION  entry  in  the  figure's  task 
network. 


This  analysis  can  be  demonstrated  via  the  sample  session 


with  the  TRIANGLE  task  in  Section  4.1.1.  The  proposed  and 
correct  orograms  for  this  task  were: 
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Analysis  of 

your  program: 

Student 

Correct 

Commentary 

Code 

Code 

DRAW  200 

DRAW  200 

Correct  mainstep  to  draw 

line  segment  AB 

TURN  60 

TURN  120 

Incorrect  interface  to  turn 

through  angle  ABC 

DRAW  200 

DRAW  200 

Correct  mainstep  to  draw 

line  segment  BC 

TURN  120 

Missing  interface  to  turn 

through  angle  BCA 

DRAW  200 

Missing  mainstep  to  draw 

line  segment  CA 


The  following  program-step  structures  are  built  from  these 

correct  and  incorrect  programs: 

(PROGRAM- STEP  1 

(T0014  DRAW  200) 

CORRECT 

(DRAW-LINE-SEGMENT  A  B) ) 

(PROGRAM- STEP  2 

(T0015  TURN  60) 

INCORRECT 

(SHOULD-BE  TURN  120) 

(TURN-THROUGH- ANGLE  ABC)) 

(PROGRAM-STEP  3 

(T0016  DRAW  200) 

CORRECT 

(DRAW-LINE-SEGMENT  B  C) ) 

(PROGRAM- STEP  4 

(T0020  TURN  120) 

MISSING-STEP 

(TURN-THROUGH- ANGLE  B  C  A) ) 

(PROGRAM-STEP  5 

(T0021  DRAW  200) 

MISSING-STEP 
(DRAW-LINE-SEGMENT  C  A)) 


R9 

In  this  analysis,  program-steps  1  and  3  identify  correct 
statements  (the  two  DRAWS).  Step  3  is  incorrect  (the  turn  should 
be  120  degrees),  and  steps  4  and  5  are  missinq. 

When  a  student  requests  a  hint,  these  pattern-step 
structures  are  matched  against  a  set  of  hint  generation 
structures  (Table  9)  that  describe  particular  programming  errors, 
such  as  the  omission  of  a  step  or  a  TURN  through  an  incorrect 
angle.  Due  to  the  constrained  nature  of  the  turtle-qraphics 
task,  the  number  of  possible  errors  is  finite  and  relatively 
small,  and  hint  generation  structures  can  be  constructed  for 
each.  The  hint  generation  structures  are  examined  secruentiallv, 
and  the  first  match  found  is  used  as  the  basis  of  a  hint; 
different  hinting  strategies  can  therefore  be  established  by 
ordering  these  structures  in  different  ways. 

TURTLES  hint  generation  rules  are  currently  ordered  so  as 
to  first  detect  and  offer  help  on  steps  missinq  from  the 
student's  program.  The  first  hint  generated  in  the  sample 
TRIANGLE  task  (Section  4.1.1)  then  focuses  on  the  absence  of  the 
final  DRAW  in  the  student's  program  —  You  are  missing  the  line 
segment  from  C  to  A.  This  hint  is  the  result  of  the  match 
between  the  program-step  structure  corresponding  to  the  missing 
DRAW; 

(PROGRAM-STEP  5 

(T0021  DRAW  200) 

MISSING-STEP 
(DRAW-LINE-SEGMENT  C  A) ) 

and  the  hint  generation  pattern; 


Table  9: 


TURTLE'S  hint  qeneration  structures,  in  the  form: 
(error-condition  =>  general-hint  specific-hint) 


Hints  for  missing  steps: 


1:  (program-step  ?  (?  draw  ?value)  missing-step  Pintei  i  i<> 


("You  are  missing  the  line  segment  from  " 
(start-node  ?intention)  "  to  " 
(end-node  ?intention)  " 

("You  are  missing  the  step  DRAW  "  ?value 
"  to  draw  the  line  segment  from  ” 
(start-node  ?intention)  "  to  " 
(end-node  ?intention)  ".") 


2:  (program-step  ?  (?  move  ?value)  missing-step  Pintention) 


=  > 


("You  are  missing  a  retrace  over  the  line  segment  frw 
(start-node  ?intention)  "  to  " 

(end-node  ?intention)  ".") 

("You  are  missing  the  step  MOVE  "  ?value 

"  to  retrace  over  the  line  segment  from  " 
(start-node  Pintention)  "  to  " 

(end-node  Pintention)  ".") 


1:  (and  ( is-cur rent-task  orientation-interface) 

(program-step  ?  (?  turn  Pvalue)  missing-step 
?  intention) ) 


("Another  TURN  command  is  necessarv  to  orient  " 
*TASK-NAME*  "  correctly.") 

("TURN  "  Pvalue  "  is  the  setup  step  necessary  to  or  ion" 
♦TASK-NAME*  "  correctly.") 


4:  (and  (not  ( is-cur rent-task  orientation-interface)) 
(program-step  ?  (?  turn  Pvalue)  missing-step 
Pintention) ) 


=> 
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("You  are  missinq  the  rotation  to  turn  through  angle  " 

(first-node  ?intention)  (second-node  ?intention) 
(third-node  ?intention) 

("You  are  missing  the  step  TURN  "  ?value  ".") 

Hints  for  extra  steps: 

1:  (and  (is-current-task  orientation-interface) 

(program-step  ?  (?  turn  ?value)  extra-step) ) 

=  > 

("Only  one  TURN  command  is  needed  to  set  up  the 
orientation  correctly.") 

("The  step  TURN  "  ?value  "  is  unnecessary.") 

2:  (and  (not  (is-current-task  orientation-interface)) 
(program-step  ?  (?  turn  ?value)  extra-step)) 

=  > 

("There  is  an  extra  TURN  command  in  your  interface.") 
("The  step  TURN  "  ?value  "  is  unnecessary.") 

3:  (and  (is-current-task  orientation-interface) 

(oroqram-step  ?  (?  move  ?value)  extra-step)) 

=  > 

("MOVE  commands  are  not  needed  in  the  orientation 
interface . " ) 

("The  step  MOVE  "  ?value  "  is  unnecessary.") 

4:  (and  (not  (is-current-task  orientation-interface)) 
(oroqram-step  ?  (?  move  ?value)  extra-step)) 

=  > 

("There  is  an  extra  MOVE  command  in  vour  interface.") 

( "The  steo  MOVE  "  ?value  "  is  unnecessary.") 


Hints  for  incorrect  steps: 

1:  (and  (is-current-task  orientation-interface) 
( is-missinq-step  turn) 

(program-step  ?  ?step  incorrect 
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(should-be  ?  ?value)  ?intention) ) 


=> 


("The  setup  rotation  for  "  *TASK-NAME*  "  is  incorrect.") 

("TURN  "  ?value  ",  not  TURN  "  (parameter  ?step) 

"  should  be  the  setup  for  "  *TASK-NAME* ) 


2:  (and  (not  ( is-current-task  orientation-interface)) 

( is-missinq-step  turn) 

(program-step  ?  ?step  incorrect 

(should-be  ?  ?value)  ’intention)) 


=  > 

("The  rotation  to  turn  through  angle  " 

(first-node  ?intention)  (second-node  ?intention) 
(third-node  ?intention)  "  is  incorrect.") 

("The  rotation  to  turn  through  angle  " 

(first-node  ?intention)  (second-node  ?intention) 
(third-node  ?intention)  "  should  be  TURN 
"  ?value  "  instead  of  TURN  "  (parameter  ?stet>) 

ff  ff  \ 


3:  (and  ( is-miss inq-step  move) 

(proqram-step  ?  ?step  incorrect 

(should-be  ?  ’value)  ?intention)) 


=  > 


("The  retrace  to  move  over  line 
(start-node  ?intention) 
"  is  incorrect.") 


segment  " 

(end-node  ?intention) 


("The  retrace  to  move  over  line  seqment  " 

(start-node  ?intention)  (end-node  ?intention) 
"  should  be  MOVE  "  ?value  "  instead  of  MOVE  " 
(parameter  ’step)  ".") 
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(PROGRAM-STEP  7 

(7  DRAW  ?  VALUE) 
MISSING-STEP 
? INTENTION) 


Position  variables  are  at  the  heart  of  this  match:  "?" 
matches  anything,  "7VALUE"  matches  any  numerical  value  (such  as 
the  extent  of  a  DRAW  or  a  TURN),  and  "7INTENTI0N"  matches  any 
intended-action  pattern,  such  as  (DRAW-LINE-SEGMENT  C  A) .  Each 
of  the  hint  generation  structures  is  associated  with  two  hints, 
one  general  and  one  specific: 


General  hint: 

("You  are  missing  the  line  segment  from  " 
(START-NODE  ? INTENTION) 

"  to  " 

(END-NODE  7INTENTI0N) 

) 

Specific  hint: 

("You  are  missing  the  step  DRAW  " 

7VALUE 

"  to  draw  the  line  segment  from  " 
(START-NODE  7 INTENTION) 

"  to  " 

(END-NODE  7INTENTI0N) 


Presenting  a  hint  to  the  student  first  requires  determining 

a 

whether  the  student  should  receive  the  general  or  the  specific 
hint.  When  a  student  first  asks  for  a  hint,  TURTLE  presents  the 
general  hint;  the  specific  hint  is  given  in  response  to  the 
second  request;  no  hints  are  given  on  subsequent  requests.  The 
selected  hint's  pattern  is  then  converted  to  the  form  the  student 
will  actuallv  see  by  merqinq  the  text  in  the  selected  hint's 
pattern  with  the  values  of  the  variables  and  the  function  calls 
embedded  within  that  pattern.  In  the  present  example,  the 
following  forms  would  be  created: 


General  hints 

You  are  missing  the  line  segment  from  c  to  A. 

Specific  hint: 

You  are  missing  the  step  DRAW  200  to  draw  the 
line  segment  from  C  to  A. 

These  proqram-steo  structures  are  also  used  to  control  the 
student's  correction  of  his  program.  After  the  student  has 
defined  his  oroqram  and  TURTLE  has  carried  out  the  analysis 
described  above,  the  student  enters  a  debugging  phase,  where  he 
is  asked  to  identify  and  correct  errors  in  his  program  (see  the 
trace  in  Section  4.1.1).  When  a  student  attempts  to  correct  one 
of  these  errors,  a  program-step  structure  is  generated  for  the 
modification.  If  this  structure  indicates  the  modification  is 
incorrect,  TURTLE  rejects  the  modification  and  tells  him  to  try 
again.  Correct  steps  cannot  be  deleted,  and  incorrect  steps  — 
typically  TURNs  with  the  wrong  angle  specified  —  must  be 
respecified  with  the  correct  value.  In  this  way,  the  student  is 
keot  working  toward  a  correct  solution  at  all  times,  although  at 
the  cost  of  preventing  the  student  from  experimenting  with 
different,  possibly  incorrect,  versions  of  his  program. 

4.3  Areas  of  Future  Development 

This  research  was  focused  on  developing  TURTLE'S  ability  to 
identify  the  plans  underlying  students'  LOGO  programs  and  to 
offer  hints  that  address  errors  in  these  olans.  This  initial 
effort  has  been  limited  in  two  wavs,  each  of  which  is  an 
important  area  for  future  research. 

The  first  of  these  is  to  remove  the  restrictions  on  the 
kinds  of  LOGO  programs  that  can  be  written;  in  particular,  to 
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allow  proqrams  with  variable  assignment,  recursion,  and 
iteration.  Fortunately,  these  components  of  LOGO  can  be 
incorporated  into  TURTLE'S  existing  system  of  task  networks  by 
modifying  and  extending  the  CONNECTIONS  section  of  a  figure's 
task  network.  Consider  the  recursive  function  "BINARY-TREE 
LENGTH  DEPTH",  which  would  draw  a  binary  tree  DEPTH  levels  deep, 
with  branches  LENGTH  units  long.  The  task  network  for  this 
figure  would  state  that,  while  some  of  the  figure's  points  are 
connected  by  MOVES  and  DRAWS,  others  are  connected  by  (recursive) 
calls  to  the  BINARY-TREE  function.  In  evaluating  a  student's 
program  for  this  figure,  TURTLE  would  have  to  determine  that  the 
student's  program  called  BINARY-TREE  at  the  appropriate  places, 
and  that  the  arguments  defining  the  length  and  number  of  the 
branches  of  the  sub-tree  were  specified  correctly.  This  process 
would  be  no  different  from  the  way  TURTLE  currently  determines 
that,  for  instance,  an  angle  is  generated  by  a  TURN  of  a 
particular  number  of  degrees;  such  an  extension  is  completely 
compatible  with  TURTLE'S  existing  representational  and  analysis 
by  synthesis  systems. 

Other  extensions  of  TURTLE  —  ones  which  reach  beyond  the 
domain  ol  turtle  graphics  —  are  possible.  However,  one  should 
not  overlook  the  advantages  of  working  within  a  highly 
constrained  domain  such  as  turtle  graphics.  TURTLE  is  successful 
largely  because  of  the  detailed  problem  representation  found  in 
the  system's  task  networks;  these  make  possible  the  generation  of 
a  set  of  possible  solutions,  one  of  which  should  correspond  to 
the  Plan  underlying  the  student's  program.  The  success  of  future 
applications  of  TURTLE'S  methodology  will  depend  upon  the 


presence  of  a  comparably  detailed  specification  of  the  new 
domain's  tasks. 

The  second  area  in  need  of  development  is  the  construction 
of  a  reasonable  pedagogical  component  for  TURTLE,  one  that  can 
develop  a  sound  model  of  the  student  and  use  this  model  to 
control  the  tutorial  process.  TURTLE  currently  lacks  a 
well-developed  curriculum  system  such  as  that  in  BIP  (Barr,  et 
al.,  1976),  in  which  the  tasks  presented  to  students  are  selected 
on  the  basis  of  the  skills  the  student  has  and  has  not  yet 
acquired;  the  task  selection  strategies  used  by  BIP  would  provide 
a  sound  beqinninq  for  this  work.  There  is  also  much  room  for 
improvement  in  turtle's  hinting  capability.  This  project  did  not 
address  the  questions  of  when  a  student  should  be  interrupted 
with  a  hint,  and  TURTLE  has  only  primitive  strategies  for 
determining  the  exact  hint  a  student  should  receive:  TURTLE 
gives  a  general  hint  when  a  hint  is  first  requested,  and  gives 
more  specific  hints  in  response  to  further  requests.  The  nature 
of  these  problems  is  discussed  in  the  summary;  for  now,  it  should 
be  noted  that  these  insufficiencies  can  be  corrected  only  by  the 
development  of  powerful  student  models,  which  will  itself  depend 
upon  careful  research  into  the  psychological  issues  underlying 
the  question  of  how  people  learn  to  program. 
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Section  5 
Summary 

The  results  of  this  research  are  encouraqinq  with  respect  to 
the  overall  feasibility  of  constructing  intelliqent  tutorial 
systems.  TURTLE  can  exploit  the  limited  structure  of  (a  modified 
version  of)  LOGO  and  the  proqramminq  problems  oresented  to 
students  to  qenerate  hints  relevant  to  a  student's  oroposed 
solution.  We  have  thus  far  run  only  informal  experiments  with 
TURTLE ,  which  have  not  been  suitable  for  a  detailed  comparison  of 
TURTLE  to  a  more  traditional  tutor  such  as  BXP.  However,  TURTLE, 
like  the  human  tutors  in  the  BXP  studies,  can  identify  the  errors 
in  a  student's  proqram  and  offer  specific  desiqn-level  and 
code-level  hints  relevant  to  these  errors.  In  addition,  TURTLE'S 
use  of  menus  and  multiple  windows  corrects  some  of  the  human 
enqineerinq  problems  observed  in  the  BIP/HINT  experiments. 

As  is  often  the  case,  however,  the  results  of  this  research 
mav  raise  more  questions  than  they  answer.  In  Darticular: 

*  what  strategies  and  knowledge  structures  are  acquired  by 
people  learning  to  program?  The  LOGO  programs  written  by  our 
subjects  typically  went  through  two  stages.  First,  the  primary 
fiqures  (trianqles,  squares,  etc.)  were  drawn  correctly,  but 
were  interfaced  incorrectly;  second,  the  interfaces  amonq  these 
fiqures  were  gradually  corrected.  These  studies  suggest  that 
people  have  good  strategies  for  drawing  regular  geometric 
figures,  but  not  for  building  the  interfaces  between  figures. 


Our  subjects'  major  difficulty  in  drawing  regular  figures 
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was  learninq  to  specify  the  figures"  angles  by  turning  the  turtle 
through  some  number  of  degrees.  Some  of  these  constructions  were 
more  difficult  to  learn  than  others:  a  90  degree  right  turn  is 
made  by  "TORN  90",  while  a  90  degree  left  turn  is  made  by  "TORN 
270";  an  equilateral  triangle  (with  equal  interior  angles  of  60 
degrees)  is  generated  by  successively  drawing  the  triangle's 
exterior  120  degree  angles.  These  were  not  major  problems  for 
our  students,  however.  After  working  through  the  TRIANGLE  task 
in  TURTLE,  they  were  almost  always  able  to  correctly  build  the 
square  and  triangular  components  of  the  more  complex  figures 
taught  by  TURTLE,  such  as  WISHING-WELL  and  NAPOLEON.  In 
contrast,  students  had  considerable  difficulty  building 
interfaces  that  interconnect  the  regular  components  of  these 
complex  figures.  Most  of  these  interfaces  were  initially 
incorrect,  and  most  of  the  time  spent  by  a  student  in  a  session 
with  TURTLE  was  devoted  to  correcting  interfaces.  It  is  clear 
that,  as  people  become  more  familiar  with  LOGO  and  turtle 
graphics,  they  become  more  able  to  build  correct  interfaces,  and 
the  study  of  the  development  of  successful  interface  strategies 
may  offer  some  valuable  insights  into  computer  programming  as  a 
cognitive  process. 

What  constitutes  a  good  hint?  When  a  tutor  has  decided  that 
a  student  needs  a  hint,  that  hint  should 

*  address  the  immediate  problem  the  student  is  having,  and 

*  create  in  the  student  a  strong  conceptual  understanding  of 
the  problem's  solution,  so  that  future  hints  will  be  unnecessary. 


These  qoals  typically  interfere  with  each  other.  The 
student's  immediate  problem  can  best  be  solved  by  simply  giving 
the  student  a  set  of  program  statements  that  will  produce  the 
desired  result;  if  the  student  simply  copies  the  provided 
solution,  however,  it  is  doubtful  he  will  retain  this  advice  for 
use  at  a  later  time.  At  the  other  extreme,  if  the  tutor  gives  a 
vaque,  qeneral  hint  about  how  the  problem  miqht  be  solved,  the 
student  will  probably  not  acquire  enough  new  information  to  allow 
him  to  either  solve  the  immediate  problem  or  increase  his 
long-term  understanding  of  the  domain.  A  good  tutorial  system 
should  provide  hints  that  address  the  specific  gaps  and  fallacies 
in  the  student's  understanding  of  the  problem.  These  errors  may, 
of  course,  occur  at  any  of  the  several  levels  of  abstraction  that 
characterize  the  student's  problem  representation,  such  as  the 
multiple  levels  of  design  and  code  structures  encountered  in  the 
programming  tasks  studied  here. 

This  issue  has  received  little  study;  it  has  been  addressed 
in  the  work  by  Collins  and  his  colleagues  on  Socratic  tutorial 
systems  (Collins,  Aiello,  Warnock,  &  Miller,  1975;  Stevens  & 
Collins,  1977).  Most  existinq  systems  try  to  deal  with  this 
problem  by  providing  two  levels  of  hints  (see  the  discussions  of 
the  OGOL  tutor,  BIP,  and  TURTLE):  a  general  hint  is  given  first 
(in  these  programming  language  tutors,  these  are  often  related  to 
the  design  of  the  program  or  individual  statements) ,  followed  by 
a  more  specific  hint  (often  showing  program  statements  similar  to 
or  the  same  as  those  that  will  solve  the  problem) .  This 
technique  only  approximates  a  tutor  that  can  target  the 
specificity  of  its  hints  to  the  needs  of  the  student.  Since  such 
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a  system  does  not  always  offer  hints  that  address  the  exact 
nature  of  a  student's  problem,  it  forces  the  student  who  needs 
fairly  specific  help  on  the  form  of  particular  statements  to  ask 
for  several  hints,  many  of  which  may  provide  irrelevant  or  even 
confusinq  information.  More  significantly,  students  can  become 
aware  of  this  aspect  of  the  tutor  very  quickly  (as  did  some  of 
the  subjects  in  our  informal  experiments  with  TURTLE),  and 
realize  that  the  exact  answers  to  their  problems  can  be  obtained 
simply  by  asking  for  several  hints.  Some  students  may  then 
deliberately  ask  for  multiple  hints  in  order  to  be  told  the 
answer  to  their  problem,  while  others  who  view  askinq  for  hints 
as  admitting  failure  (see  Section  3.4.4)  may  avoid  hints 
altogether  for  fear  of  beinq  given  a  hint  that  reveals  the  exact 
answer.  Beinq  able  to  qenerate  a  truly  appropriate  hint  for  a 
particular  student  in  a  particular  task  is  thus  a  critical 
component  of  a  truly  successful  tutor,  one  that  will  depend  upon 
understanding  the  psychological  components  of  the  instructional 
and  learning  experiences,  and  incorporating  these  components  into 
an  accurate  model  of  the  student  and  the  tutorial  environment. 
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Appendix  A: 

TURTLE  ' s  implementation 

12 

TURTLE  is  implemented  in  ASSERT,  a  CONNIVER-like 
assert ional  database  system  implemented  in  MACLISP.  It  uses  a 
standard  computer  terminal  for  input  and  the  display  of  TURTLE'S 
Droqram  analysis,  and  a  high  resolution  color  graphics  terminal 
(512  x  512  pixels)  for  the  display  of  LOGO  figures. 

Durinq  the  task  solution  and  assistance  process,  TURTLE  uses 
two  graphics  windows  and  one  text  window  on  the  graphics 
terminal.  The  graphics  windows  display  the  figure  a  correct 
program  should  draw  and  the  figure  drawn  by  the  student's  current 
program.  These  two  windows  occupy  the  left  half  of  the  screen  on 
the  graphics  terminal.  A  text  window  on  the  right  half  of  the 
screen  window  displays  program  definitions  and  modifications, 
hints,  and  tutorial  text.  TURTLE  can  be  run  without  a  graphics 
terminal,  although  a  diagram  of  the  four  tasks  will  be  required 
to  follow  turtle's  operation. 

TURTLE'S  response  time  varies,  but  is  usually  under  3 
seconds  for  requests  for  hints,  program  edits,  text  display,  and 
similar  operations.  Open  coded  solutions  can  take  more  time, 
depending  on  the  complexity  of  the  figure  to  be  analyzed  (an  open 
coded  solution  for  the  tree  task  with  one  or  zero  retraces  can  be 
recognized  in  15  seconds  or  less) . 

— 

ASSERT  was  designed  by  Mark  Miller  and  William  Murray,  and 
implemented  by  Murray. 
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Navy 


1  Psychologist 

ONR  Branch  Office 
1030  East  Green  Street 
Pasadena,  CA  91101 

1  Special  Asst,  for  Education  and 
Training  (OP-OIE) 

Rm.  2705  Arlington  Annex 
Washington,  DC  20370 


Navy 


1  Dr.  Alfred  F.  9a ode 

Training  Analysis  A  Evaluation  Group 
(TAEG) 

Dept,  of  the  Navy 
Orlando,  FL  32813 

1  Dr.  Richard  Sorensen 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 


1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  &  Studies  Branch 
(OP-115) 

Washington,  DC  20350 


Roger  Weisslnger-Baylon 
Department  of  Administrative  Sciences 
Naval  Postgraduate  School 
Monterey,  CA  93940 


1  LT  Frank  C.  Petho,  MSC,  USN  CPh.D) 

Selection  and  Training  Research  Division 
Human  Performance  Sciences  Dept. 

Naval  Aerospace  Medical  Research  Laborat 
Pensacola,  FL  32508 

1  Dr.  Gary  Poock 

Operations  Research  Department 
Code  55PK 

Naval  Postgraduate  School 
Monterey,  CA  93940 


1  Dr.  Robert  Wisher 
Code  309 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Mr  John  H.  Wolfe 
Code  P310 

U.  S.  Navy  Personnel  Research  and 
Development  Center 
San  Diego,  CA  92152 


1  Roger  W.  Remington,  Ph.D 
Code  L52 
NAMRL 

Pensacola,  FL  32508 

1  Dr.  Bernard  Rimland  (03B) 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

1  Dr.  Worth  Scanland,  Director 

Research,  Development,  Test  &  Evaluation 
N-5 

Naval  Education  and  Training  Command 
NAS,  Pensacola,  FL  32508 


Dr.  Robert  G.  Smith 

Office  of  Chief  of  Naval  Operations 

OP-987H 

Washington,  DC  20350 
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Army 


Ar»y 


1 


Technical  Director 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


Dr.  Frederick  Steinheiser 

Dept,  of  Navy 

Chief  of  Naval  Operations 

OP-113 

Washington,  DC  20350 


1  Mr.  James  Baker 

Systems  Manning  Technical  Area 
Army  Research  Institute 
5001  Eisenhower  Ave. 
Alexandria,  VA  2233? 


1  Dr.  Joseph  Ward 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


1  Dr.  Beatrice  J.  Farr 

U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22?3? 


1  DR.  FRANK  J.  HARRIS 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

1  Dr.  Michael  Kaplan 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA.  VA  2233? 


1  Dr.  Milton  S.  Katz 

Training  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 
Attn:  PERI-OK 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22?5? 


1 


Dr.  Robert  Sasmor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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Air  Force 


Marines 


1  U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Rolling  Air  Force  Base 
Washington,  DC  20332 

1  Dr.  Genevieve  Haddad 
Program  Manager 

Life  Sciences  Directorate 
AFOSR 

Bolling  AFP,  DC  2033? 

2  3700  TCHTW/TTGH  Stop  32 
Sheppard  AFB,  TX  7f’?11 


1  H.  William  Greenup 

Education  Advisor  (E031) 
Education  Center,  MCDEC 
Quantico,  VA  22134 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington,  VA  22217 

1  DR.  A.L.  SLAFK0SKY 

SCIENTIFIC  ADVISOR  (CODE  RD-1 ) 
HO,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20?PO 
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Coast Guard  Other  DoD 


1  Chief,  Psychological  Feserch  Branch  12  Defense  Technical  Information  Center 

U.  S.  Coast  Guard  (G-P-1/2/TP42)  Cameron  Station,  Bldg  5 

Washington,  DC  20593  Alexandria,  VA  22314 

Attn:  TC 

1  Military  Assistant  for  Training  end 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defense 
for  Research  A  Engineering 
Room  3D129,  The  Pentagon 
Washington,  DC  20301 

1  DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 
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Civil  Govt 


1  Dr.  Susan  Chipman 

Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  John  Mays 

National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  William  J.  McLaurin 
86610  Howie  Court 
Camp  Springs,  MD  20031 

1  Dr.  Arthur  Melmed 

National  Intitute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Molnar 
Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Joseph  Psotka 

National  Institute  of  Education 
1200  19th  St.  NW 
Washington, DC  20209 

1  Dr.  Frank  Withrow 

U.  S.  Office  of  Education 
*100  Maryland  Pve.  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  A  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 


Non  Govt 


1  Dr.  John  R.  Anderson 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Anderson,  Thomas  H. ,  Ph.D. 

Center  for  the  Study  of  Reading 
17*1  Children's  Research  Center 
51  Gerty  Drive 
Champiagn,  TL  61820 

1  Dr.  John  Annett 

Department  of  Psychology 
University  of  Warwick 
Coventry  CVH  7AL 
ENGLAND 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

1  Dr.  Alan  Baddeley 

Medical  Research  Council 

Applied  Psychology  Unit 
15  Chaucer  Road 
Cambridge  CB2  2EF 
ENGLAND 

1  Dr.  Patricia  Baggett 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Mr  Avron  Barr 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  9**305 

1  Liaison  Scientists 

Office  of  Naval  Research, 
Branch  Office  ,  London 
Box  39  FP0  New  York  09510 

1  Dr.  Lyle  Bourne 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 
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Non  Govt 


Non  Govt 


1  Dr.  John  S.  Brown  1 

XEROX  Palo  Alto  Research  Center 
33? 3  Coyote  Road 
Palo  Alto,  CA  99309 

1  Dr.  Bruce  Buchanan  1 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  99305 

1  DR.  C.  VICTOR  BUNDERSON  1 

WICAT  INC. 

UNIVERSITY  PLAZA,  SUITE  10 
1180  SO.  STATE  ST. 

OREM,  UT  89057 

1  Dr.  Pat  Carpenter  1 

Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  John  B.  Carroll  1 

Psychometric  Lab 
Univ.  of  No.  Carolina 
Davie  Hall  013A 
Chapel  Hill,  NC  27519 

1 

1  Dr.  William  Chase 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1 

1  Dr.  Micheline  Chi 

Learning  RAD  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 

Pittsburgh,  PA  15213  1 

1  Dr.  Allan  M.  Collins 

Bolt  Beranek  A  Newman,  Tnc. 

50  Moulton  Street 

Cambridge,  Ma  02138  1 

1  Dr.  Lynn  A.  Cooper 
LRDC 

University  of  Pittsburgh  1 

3939  O'Hara  Street 
Pittsburgh,  PA  15213 


Dr,  Meredith  P.  Crawford 
American  Psychological  Association 
1200  17th  Street,  N.W. 

Washington,  DC  200?6 

Dr.  Kenneth  B.  Cross 
Anacapa  Sciences,  Inc. 

P.0.  Drawer  Q 

Santa  Barbara,  CA  93102 

LCOL  J.  C.  Eggenberger 

DIRECTORATE  OF  PERSONNEL  APPLIED  RESEARC 

NATIONAL  DEFENCE  HQ 

101  COLONEL  BY  DRIVE 

OTTAWA,  CANADA  K1A  0K2 

Dr.  Ed  Feigenbaum 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  99305 

Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.0.  Box  188 

Iowa  City,  IA  52290 

Mr.  Wallace  Feurzeig 
Bolt  Beranek  A  Newman,  Inc. 

50  Moulton  St. 

Cambridge,  MA  02138 

Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

Univ.  Prof.  Dr.  Gerhard  Fischer 
Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

DR.  JOHN  D.  FOLLEY  JR. 

APPLIED  SCIENCES  ASSOCIATES  INC 
VALENCIA,  PA  16059 

Dr.  John  R.  Frederiksen 
Bolt  Beranek  A  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 
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Non  Govt 


Non  Govt 


1  Dr.  Alinda  Friedman  1 

Department  of  Psychology 
University  of  Alberta 
Edmonton,  Alberta 
CANADA  T6G  2E9 

1 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 

PITTSBURGH,  PA  15213  1 

1  Dr.  Marvin  D.  Glock 
21?  Stone  Hall 
Cornell  University 

Ithaca,  NY  14853  1 

1  Dr.  Daniel  Gopher 

Industrial  &  Management  Engineering 
Technion-Israel  Institute  of  Technology 
Haifa  1 

ISRAEL 

1  DR.  JAMES  G.  GREENO 

LRDC  1 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1 

1  Dr.  Ron  Hambleton 
School  of  Education 
University  of  Massechusetts 
Amherst,  MA  01002 

1 

1  Dr.  Harold  Hawkins 

Department  of  Psychology 
University  of  Oregon 
Eugene  OR  97 403 

1 

1  Dr.  Barbara  Hayes-Roth 
The  Rand  Corporation 
1700  Main  Street 
Santa  Monica,  CA  90406 

1 

1  Dr.  Frederick  Hayes-Roth 
The  Rand  Corporation 
1700  Main  Street 
Santa  Monica,  CA  90406 


Dr.  James  R.  Hofftnan 
Department  of  Psychology 
University  of  Delaware 
Newark,  DE  19711 

Dr.  Kristina  Hooper 
Clerk  Kerr  Hall 
University  of  California 
Santa  Cruz,  CA  95060 

Glenda  Greenwald,  Ed. 

"Human  Intelligence  Newsletter" 
P.  0.  Box  1163 
Birmingham,  MI  48012 

Dr .  Earl  Hunt 
Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 

Dr.  Ed  Hutchins 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152 

DR.  KAY  INABA 
21116  VANOWEN  ST 
CANOGA  PARK.  CA  9130? 

Dr.  Steven  W.  Keele 
Dept,  of  Psychology 
University  of  Oregon 
Eugene,  OR  97403 

Dr.  Walter  Kintsch 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80302 

Dr.  David  Kieras 
Department  of  Psychology 
University  of  Arizona 
Tuscon,  AZ  85721 

Dr.  Stephen  Kosslyn 
Harvard  University 
Department  of  Psychology 
3?  Kirkland  Street 
Cambridge,  MA  02138 
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Non  Govt 


1  Dr.  Marcy  La n am an 

Department  of  Psychology,  NT  25 
University  of  Washington 
Seattle,  WA  98195 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  Lesgold 
Learning  HAD  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  15260 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Erik  McWilliams 

Science  Education  Dev.  and  Research 
National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Mark  Miller 

TI  Computer  Science  Lab 
C/0  2829  Winterplace  Circle 
Plano,  TX  75075 

1  Dr.  Allen  Munro 

Behavioral  Technology  Laboratories 
1895  Elena  Ave.,  Fourth  Floor 
Redondo  Beach,  CA  90277 

1  Dr.  Donald  A  Norman 

Dept,  of  Psychology  C-009 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 


Ido  Govt 


1  Committee  on  Human  Factors 
JH  811 

2101  Constitution  Ave.  NW 
Washington,  DC  20918 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
900  Army  Navy  Drive 
Arlington,  VA  22202 

1  Dr.  Seymour  A.  Papert 

Massachusetts  Institute  of  Technology 
Artificial  Intelligence  Lab 
595  Technology  Square 
Cambridge,  HA  02139 

1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  Dr.  James  W.  Pellegrino 
University  of  California, 

Santa  Barbara 
Dept,  of  Psychology 
Santa  Barabara,  CA  93106 

1  MR.  LUIGI  PETRULLO 

2931  N.  EDGEWOOD  STREET 
ARLINGTON,  VA  22207 

1  Dr .  Martha  Pol  son 

Department  of  Psychology 
Campus  Box  396 
University  of  Colorado 
Boulder,  CO  80309 

1  DR.  PETER  POLSON 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
BOULDER,  CO  80309 

1  Dr.  Steven  E.  Poltroek 
Department  of  Psychology 
University  of  Denver 
Denver, CO  80208 
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Non  Govt 


Non  Govt 


1  M INRAT  M.  L.  RAUCH 
P  II  4 

BUNDESMINISTERIUM  DER  VERTEIDIGUNG 

POSTFACH  1328 

D-53  BONN  1,  GERMANY 

1  Dr.  Fred  Relf 
SESAME 

c/o  Physics  Department 
University  of  California 
Berkely,  CA  94720 

1  Dr.  Lauren  Resnlck 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Mary  Riley 
LRDC 

University  of  Pittsburgh 
?939  O'Hara  Street 
Pittsburgh,  PA  1521? 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 


1  DR,  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRRO 

300  N.  WASHINGTON  ST. 
ALEXANDRIA,  VA  2231** 

1  Committee  on  Cognitive  Research 
%  Dr.  Lonnie  R.  Sherrod 
Social  Science  Research  Council 
605  Third  Avenue 
New  York,  NY  10016 

1  Robert  S.  Siegler 
Associate  Professor 
Carnegie-Mellon  University 
Department  of  Psychology 
Schenley  Park 
Pittsburgh,  PA  15213 

1  Dr.  Edward  E.  Smith 

Bolt  Beranek  A  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Robert  Smith 

Department  of  Computer  Science 

Rutgers  University 

New  Brunswick,  NJ  08903 


1  Dr.  Ernst  2.  Rothkopf 
Eel]  Laboratories 
600  Mountain  Avenue 
Murray  Hill,  NJ  07974 


1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 


1  Dr.  David  Rumelhart  1 

Center  for  Human  Information  Processing 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  DR.  WALTER  SCHNEIDER 

DEPT.  OF  PSYCHOLOGY  1 

UNTVERSTY  OF  ILLINOIS 
CHAMPAIGN,  TL  61820 

1  Dr.  Alan  Schoenfeld 

Department  of  Mathematics  1 

Hamilton  College 
Clinton,  NY  1332? 


Dr .  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

DR.  ALBERT  STEVENS 
BOLT  BERANEK  &  NEWMAN,  INC. 
50  MOULTON  STREET 
CAMBRIDGE,  MA  02138 

David  E.  Stone,  Ph.D. 

Hazel  tine  Corporation 
7680  Old  Spr inghouse  Road 
McLean,  VA  22102 
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Non  Govt 


1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  99305 

1  Dr.  Klkuroi  Tatsuoka 

Computer  Eased  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbann,  IL  61  SOI 

1  Dr.  John  Thomas 

IBM  Thomas  J.  Watson  Research  Center 
P.0.  Box  218 

Yorktown  Heights,  NY  10598 

1  DR.  PERRY  THORNDYKE 
THE  RAND  CORPORATION 
1700  MAIN  STREET 
SANTA  MONICA,  CA  90*106 

1  Dr.  Douglas  Towne 

Univ.  of  So.  California 
Pehavioral  Technology  Labs 
13*15  S.  Elena  Ave. 

Redondo  Beach,  CA  90277 

1  Dr.  J.  Uhlaner 

Perceptronics,  Tnc. 

6271  Variel  Avenue 
Woodland  Hills,  CA  91?6*» 

1  Dr.  Benton  J.  Underwood 
Dept,  of  Psychology 
Northwestern  University 
Evanston,  JL  *0201 

1  Dr.  David  J.  Weiss 
N660  Elliott.  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MM  55**55 

1  DR.  GERSH0N  WFLTMAN 
PERCEPTRONICS  INC. 

627 1  VARIEL  AVE. 

WOODLAND  HTLLS ,  CA  91367 


Dr.  Keith  T.  Weseourt 
Information  Sciences  Dept. 
The  Rand  Corporation 
1700  Main  St. 

Santa  Monica,  CA  90906 


